Speed up repetative measurements with NIDAQmxBase

[sth]

This is sort of a continuation of a previous thread, but it drifted enough.

I am using NIDAQmx BASE to make some repetative measurements. Briefly I want to grab N samples from a list of channels and average them every second. N is is somewhere such that I can get a good noise reduction but less than 1 second.

I set up a DAQ task set the timing and then start and read the task in a loop, at the end of the loop I kill the task.

The performance is miserable. And the data is bad. The problem is that the start task reconfigures the board each time around the loop. This means that there is some settling as data is first taken on those channels AND the set up time is very long (100s of msec). Under traditional daq I could set up an AI task configure it and then repetatively request N samples.

It seems to need that the task is completely reconfigured each time it is started. Anyone have a good idea as to how to tweak this so I can avoid all the reconfiguration?

[Dennis_Knutson]

Where is your Start Task function? It should be outside the loop. It only needs to be called once.

[AlanArmstead]

Dennis is right on....you can change your task to be continuous, then start the task once, and only read in your loop.  Here's the modifications to your test VI.

-Alan A.

[sth]

I haven't looked at the example, yet. BUT I am fairly sure from your message that this won't satisfy my needs. This is a high availability program. Thus it must be immune to varying processor loads, network storms, OS oddities, instrument hangups on serial, gpib busses, etc. Thus continuous DAQ will have a problem and overflow a buffer at some point. Thus I want only a finite data sample.

Thus I only want it to take a buffer of data at a certain time and then retrieve it. I could do continuous ACQ IFF there was some way to tell it to ignore buffer overflow. Thus at a point in time I could retrieve the last 100 to 500 msec of data, this would be great! I have never been able to do this under traditonal DAQ (up thru 6.7) and with NIDAQmx Base.

Is there a way to set up a continuous ACQ and ignore all buffer overflows, but sample the most recent chunk of data? Under LV 2.2.1 I could do this. But when the DAQ interface was rewritten for LV 3, I have never been able to set it up again. I admit that DAQ is a lot faster and the interface is much better. I like the task based interface of NIDAQmx Base but this ability is something I haven't been able to implement. It seems like a nice function to have. (IMHO)

I want to be able to handle any amount of processor overload and pickup reliably at some time in the future. I can set the DAQ buffer size big, but it is always how big? I can make it 10 minutes, I can make it 30 minutes, but I would like NEVER to have such a limit in the code.

[AlanArmstead]

You might want to consider something like doing a continuous acquisition (continuous reads) but discard or simply ignore all the data points you don't care about.  It shouldn't be too processor intensive (especially if the data acq is performing DMA transfers, which is does by default in NI-DAQmx Base), but this of course will be system dependent.  There are a couple of ways to handle this in LabVIEW, like a producer-consumer loop, or simply using globals (might need to use semaphores/rendezvous to protect shared data structures). 
-Alan A.

[sth]

yes, the consumer producer model would resolve conflict WITHIN LV on CPU scheduling. This would allow things in LV to leave a small amount of time for the data read to happen. BUT I am thinking on the whole system/OS level. Certain things like power outages during a network transfer sometimes lock up the system in the kernel. Sometimes other processes do not play nicely. Now most of the problems in the past were with "Cooperative" multitasking. Now with modern "Premptive" multitasking this should not be as much of a problem.

Actually having the DMA just dump an endless buffer in a loop into memory would be useful. Then the ability to go in and retrieve data relative to the last item inserted would allow what I want. Unfortunately this would be a redesign of the NIDAQmx architecture. I do have a copy of NBHI4.7/488V3.7/NIDAQ4.65.sea but finding hardware that will run that version is a bit difficult!! 🙂

Since I have enough CPU power at the moment I may try to do a continuous acquistion though since I am switching things I need to carefully synchronize the retreival and discard parts of the data. I just did not want to re-engineer a fairly old program. Oh well.

-Scott

[Jarrod_B.]

Scott,

Unfortunately, you cannot configure a DAQmx Base task to overwrite unread samples in the buffer.  You can do this in DAQmx using the DAQmx Read Overwrite Mode property, so if you want to develop your application on either Windows or Linux, this would be an option, but I'm still not sure that it's the best way of approaching this.  If you are developing on a Mac, the previous suggestion about using the producer/consumer loops would be your best option.

I hope this information helps.

Best regards,

[sth]

And.... The answer is a resounding NO!! It seems that it is impossible to synchronize a repetative finite sampling with an external asynchronous event. I decided to set up a continuous acq and then after my event I would read the current buffer and dump it and then read my 0.5 sec of data. That way I would not need to restart/reconfigure the input. This would get rid of the glitch that is always in the data immediately after a start/reconfigure function.

BUT guess what! The "retrieve all samples" item of specifying -1 as the number of samples to read does not retrieve all the samples that are in the buffer but rather the fixed number of samples you specified when configuing the DAQ. Thus trying to read and dump to flush the accumulated samples is not possible. This Retrieve all is somewhat misleading in the documentation. The definition of "all" is not what one would expect. So I just spent this morning ripping out what I had changed and putting it back to the finite sample method. The consumner-producer model is not good since I have to synchronize the reads with the switching of the GPIB multiplexor or at least synchronize what data to discard and what to keep. This would be cumbersome if not unreliable in the consumer producer model.

Thus emptying the buffer seems to be a somewhat cumbersome and unreliable operation involving reading and timeing the read to see if it waits for more data. This is not good since the timing is vastly variable depending on other system performance.

The other thing that would work, is to have the read finite samples actually start the read. The DAQmxStart should not do a configure as well. That should be in the task setup. Start should merely trigger acquistion of a finite number of data points for the already configured setup. Conversely there could be a pause/resume for acqusition as well but that would not be as deterministic.

[AlanArmstead]

Just to clear something up, I couldn't find anywhere in the DAQmx Base help that a -1 for the "number of samples per channel" input to the DAQmx Base Read VI would read all samples available.  That is what it does in DAQmx, but not DAQmx Base.  If you did read that for DAQmx Base, please let me know where, so we can get that documentation corrected. 

As far as your application, I'm not exactly sure why a producer/consumer loop would not work for you.  Effectively, it could be treated exactly as if you were doing a continuous data acquisition and over-writing your buffer.  The only difference is that your data acquisition VI would be controlling your buffer instead of the DAQmx Base driver (buffer is in user space and not driver space).  So, your daq VI is just continuously reading and writing those points to an array, constantly overwriting the old data.  Then, just read from the array whenever you like.  You would probably need to protect that data structure with a semaphore (or rendezvous, etc.) to ensure that you weren't reading and writing at the same time.  You could expand this to be as elaborate as you want, i.e. if you want to grab 500 samples at a time, you could have your buffer be 1000 samples and maintain a current index integer.  That way, you could grab "pre-trigger" data.

Finally, because of the flexibility of NI-DAQmx Base, you could probably create your own start VI that only contains a couple of the low-level VIs that actually start the hardware.  Then, use only that VI inside your loop.  (I haven't tried this, so I'm not exactly sure how/if this will work)  Potentially, this could speed up the execution of the "start".

-Alan A.

[sth]

Now I can't find the label... It was a labeled input where the label had a -1 for all points. It may have been left over from a previous version of LV where my actual application was built and the label propagated into my own code and was kept there as I updated. I will look for it again.

OK, it was a left over label from a previous verison in my code "Number of Scans (all:-1)". However the default value for this number of samples/channel is -1. The documentation for NI-DAQmx Base Read is incomplete. It says that the meaning of this input depends on if there is a finite or continous acquisition going on. If it is finite then the number of samples is the number configured. But there is no description as to how many samples are acquired for a continouse acquisition. It seems that the number of samples returned is NOT dependent on if the acquistion is continuous or not but is always the number of samples configured if the input is -1.

The consumer/producer format: I can not guarantee in this application that there will be enough CPU power to serve the producer part of the loop. This application is HEAVILY Multithreaded with DAQ, digital I/O, GPIB, Serial, network service and a UI. Some of this other I/O can cause the entire thread to lockup. I want it to degrade gracefully as CPU usage can spike up very high in certain sections. In this case the producer part of the loop will not empty the buffer to a labview circular buffer before overflowing. I have tried making the buffer MUCH larger (ie order of magnitude) than the acquisition but I still overflow the buffer occasionally. I could detect the error and restart the acquisition in the producer loop or make the buffer 2 orders of magnitude larger, but they both seem to be kludges.

This application was so heavily threaded that it exposed several bugs in the serial I/O drivers and VISA in the past. The last idea is probably the best, but I am trying not to invest that much time in updating this application. I have rearranged some of the logic a bit and the performance with the reconfiguration of the analog input is now acceptable. Thanks for all your advice on this.