topic Re: Deterministic loop is delayed when continuously rewriting AO's buffer in LabVIEW

Deterministic loop is delayed when continuously rewriting AO's buffer

sls__ — Tue, 23 Jul 2013 15:42:37 GMT

Hello everyone!

I am using the RT module for neurofeedback: Analyzing brain activity and then giving the user auditory feedback about it.
My RT target is a desktop pc and I use the typical deterministic + nondeterministic loop structure.

What I want is to continuously output a pure tone. The amplitude of the tone is calculated in real time and depends on some feature of the user's brain activity. It should vary smoothly to avoid click-like artifacts.

Since there are no sound VIs in the RT module, I use the DAQmx write VI (analog output) with regeneration enabled. The sample mode is continuous.

Before the first iteration, I generate a sine wave of an appropiate length (some hundred samples) and amplitude 1. Then, on each iteration, I compute the new value of the amplitude and multiply the sine wave with a linear ramp so that I get a smooth amplitude modulation. On the same iteration I write this new piece of data with the DAQmx write VI.

WIth this approach my timed loop gets delayed periodically, even thought the cpu load stays below 20%. It seems that continuously rewriting the device's buffer is the problem.

Is there a cleverer way of doing what I want? For instance: Could I write the pure tone only in the beginning and then control the amplitude by setting a gain? I have seen that a gain property exists ( http://zone.ni.com/reference/en-XX/help/370469AA-01/daqmxprop/attr118/ ) but I fail to find it. I am using LV 9.

I will be very grateful for any ideas!

Thank you!

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

sls__ — Wed, 24 Jul 2013 08:15:23 GMT

ok, the short version of the above:

I have an AO task in continuous sample mode. Regeneration is enabled. In a timed loop of period ~10 ms I use the DAQmx write VI to update the buffer on each iteration. This doesn't produce noticeable CPU loads, yet the timed loop is periodically delayed.

Is there a better/faster/alternative way to update the AO buffer?

Hopefully someone has an idea... Thanks!

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

sls__ — Wed, 24 Jul 2013 08:48:48 GMT

... and a simplified VI reproducing the problem:

After about 5s the loop shows its first delay.

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

BRennhofer — Tue, 30 Jul 2013 14:36:30 GMT

I think the problem are the buffer on the daq device.

From LV - Help:

Output Tasks

For generations, the amount of data you write before starting a generation determines the size of the buffer. The first call to a Multiple Samples version of the Write function/VI creates a buffer and determines its size.

You also can use the Output Buffer Config function/VI to create an output buffer. If you use this function/VI, you must use it before writing any data.

The samples per channel attribute/property on the Timing function/VI does not determine the buffer size for output. Instead it is the total number of samples to generate. If n is your buffer size, setting samples per channel to 3×n generates the data in the buffer exactly three times. To generate the data exactly once, set samples per channel to n.

NI-DAQmx does not create a buffer when the sample mode on the Timing function/VI is set to Hardware Timed Single Point.

To your example. I would connect the timing source from my daq device to the timed loop, so both, the DAQ device and the timed loop have the same source.
And I located manually the buffer. In your first example the buffer was only 300 samples big. I made a buffer that was 6 times greater.

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

sls__ — Tue, 30 Jul 2013 17:38:52 GMT

Hey Rennhofer, thanks for the helpful suggestions!

My guess was that a small buffer should suffice, since I would never write more than those 300 points. Could you explain to me what is the advantage of a bigger buffer?

I get a problem opening your vi, it tells me it is version 13.0 (didn't know it was out already!) Could you upload it again in version 12? Thanks!

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

The_Seeker — Tue, 30 Jul 2013 18:02:42 GMT

Hi sls,

The advantage of a larger buffer size is a lower frequency of buffer updates required for the same time interval. (Of course, the down side is that you have a reduction in the granularity of control over the waveform.)

If you are experiencing performance problems, a larger buffer size is a good place to start (-- assuming your application can tolerate a slower change of the waveform output). While processor speed is a good indicator of system throughput, it isn't always the only consideration. Loading the AO buffer repeatedly may not require max out your average CPU throughput, but there may be a burst throughput requirement that you are not meeting with your current configuration.

If I were you, I would start by bumping up the buffer size and seeing if that helps. If so, then you can inch it back down to find the best throughput/performance compromise.

BTW, I once worked on a high-speed robot & vision application installed on a custom-built quad-core Xeon system running RT. Although the 4 processors were running at less than 40% most of the time, we were never able to meet the throughput requirements. After upgrading to 8 cores, processor load dropped quite dramatically. Peak load was occassionally around 10% on some processors, but generally less than 5% most of the time. After the upgrade, we were able to meet the throughput requirements, and even had enough overhead to "beef up" the analysis algorithms. The moral of the story: processor load is an "averaged" indication of what is going on under the hood. If your system cannot keep up with the peak loads required by your application, you may experience difficulties. (At least, that was my take-away from the experience...)

Anyway, best of luck sorting out this issue.

--Dave

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

sls__ — Fri, 09 Aug 2013 17:09:01 GMT

@Seeker: Nice to know that I am not the only one experiencing strange behaviours!

Thanks both of you for the helpful comments!

Expanding the buffer has indeed helped solving the problem. It works using a buffer 6-10 times larger than the signal size and setting the "Do not allow regeneration" property. Allowing regeneration slowly consumes the free space. Yeah! Now the available buffer space oscillates at high values - Thanks for the slide, Rennhofer.

... But I don't get why it works. The update frequency is just like before, so I would expect the same load due to writing. I guess that the write VI does something extra when the buffer is getting full?

I have another question! Later in my original code I need to stop the signal generation and restart the task after a while. If I try that, I get an error -200290 (generation stopped) when I try to write to the buffer again. It doesn't happen if I do allow regeneration - but then my first problem would be back. Any suggestions on how to deal with this?

(I can think of two workarounds that I would prefer to avoid: Outputting an array of zeros while the program pauses or using a huge buffer size with regeneration on.)

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

BRennhofer — Sat, 10 Aug 2013 19:13:08 GMT

Hi sls

I'm back in the office on monday. So I will take care your question and VI . If I found a solution I will write back.

Sorry for my first VI I was uploaded in LV 2013. I'm an AE at NI so I got a prereleas of LabVIEW.

Kind regards
Bernhard

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

The_Seeker — Sat, 10 Aug 2013 22:57:24 GMT

Hi sls_

Your comment:

"I guess that the write VI does something extra when the buffer is getting full?"

My guess is that multiple calls are being made to the memory manager, requesting more memory. As I understand it, LV asks the memory manager to allocate a chunk that is "expected to be adequate". If it needs more, it has to go back again.

If you have allocated a buffer 6-10 times larger than the signal size, and you aren't regenerating, I'm guessing you are avoiding subsequent memeory manager calls (which can be slow). Apparently, when regenerating, it would appear that the memory manager may be called repeatedly.

This would explain what you are seeing -- although only the LabVIEW development team knows for sure...

Best of luck!

-- Dave

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

BRennhofer — Mon, 12 Aug 2013 07:55:17 GMT

Hi,

I agree with seeker and his arguments. But to be sure we have to discuss with the NI R&D Team.

To the error -200290. You could use the regeneration mode and use following settings:

With this settings the date in the onboard memory is allways filled and you have also enough speed while the transfer is directly above the DMA. But attention! You have only 8 DMAs on a PC system.

kind regards
Bernhard

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

BRennhofer — Mon, 12 Aug 2013 07:56:59 GMT

some interest links:

Configuring the Data Transfer Request Condition Property in NI-DAQmx

http://digital.ni.com/public.nsf/allkb/45A7AC6B59E026B386256F90006DAA49

Data Transfer Request Condition for Continuous Analog Output using NI-DAQmx

http://digital.ni.com/public.nsf/allkb/45A7AC6B59E026B386256F90006DAA49

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

sls__ — Mon, 12 Aug 2013 16:06:36 GMT

Dankeschön, Bernhard (I guess you are german 🙂 ?

Those links were very explanatory. I am still somewhat lost, but we're getting closer to solving this.

Your suggested parameters work! In the simplified VI as well as in the original VI. Unfortunately, the first iterations output the same signal once or several times. Additionally, if I call the DAQmx Stop Task VI and restart the task after a while, the first piece of signal that comes out belongs to the old update.

I can avoid this by using the following configuration: no regeneration, DMA, condition = onboard memory empty. The available space still gets consumed, but only very slowly, so it's OK. BUT: If I use this configuration in my original code, then things get weird, and I get errors -200290, -200018 and -200016 depending partly on the sampling frequency chosen, and partly on things that I haven't been able to track down.

Actually, to be more precise: In the original code, the AO is generated in a first round without problems. Then, a pause follows, and the AO is generated again. This time there is more computation going on in parallel, but I don't feel that it is so much that the computer couldn't handle it - in fact, the timed loop doesn't get delayed anymore.

My confusion arises from the fact that:

- Regeneration works fine (though I haven't checked the latency yet) in the case where more computation is going on. However, it produces bad first iterations, regardless of the computational load.

- No regeneration fails in the case with more computation, but the first iterations (when it works) are as desired.

So the computer can solve the two pieces of what I need separately. Is there a way of getting the best of the two options?

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

sls__ — Tue, 13 Aug 2013 10:07:36 GMT

Sorry, that was wrong! I had a mistake in the original VI and I wasn't monitoring all missed periods. I would like to upload that VI, but it is a true monster mammoth...

So, the things look like this: In the original VI, I start the task once (phase I), stop it after some seconds, and restart it again, this time with more computation going on in parallel (phase II). It is nevertheless not so much computation we're talking about.

All configurations work fine with phase I.

In phase II, the only configuration not delaying the timed loop or throwing an error is Bernhard's (DMA, reg, memory half full or less). The problem: Since the onboard FIFO is more than half full, the VI outputs always a piece of old data when restarted. I also haven't measured the latency, but I expect some.

I tried to set the transfer condition to "onboard memory empty" to avoid having old data played at the beginning. The timed loop then gets delayed in the first iteration or couple of iterations, the VI throws a warning 200015 and then works fine until the end.

If I disable regeneration, no matter what the transfer condition is, the timed loop is delayed, an error -200290 is thrown, and nothing is output anymore.

So, this makes some more sense now. In the line same of what The_Seeker commented, it seems that the write VI produces a peak just at the beginning of phase II and behaves fine from then on. Setting the condition to half full or less takes care of this peak, but leaves the buffer half full with old data. Mmmh....

Re: Deterministic loop is delayed when continuously rewriting AO's buffer

sls__ — Thu, 15 Aug 2013 14:41:20 GMT

Solved!

This one was subtle.

The first iteration calls the Write VI after a time t (black line). It loads a full period of data into the board's buffer. The board will be busy until time t after the start of the next iteration (red line).

Now, since the next iterations contain a little bit more computation, they will need t' > t until they can call the Write VI. By then, the board's buffer is already empty and the AO has crashed.

To solve the issue, one has to make sure that t is as large as possible. This way, the following iterations will have plenty of time to load their data into the baord.

I came with this solution:

No regeneration and transfer condition = memory empty. Fast and with controlled* latency, just the way I wanted it 🙂 But it doesn't look too elegant, it blocks execution and it may be problematic if the contents of the middle frame change substantially. Is there any other way to tell the timed loop to run the Start Task VI just before the end of the iteration?

Thanks to Bernhard and Seeker for their helpful comments!

*as good as it gets, I guess!