acquiring and plotting large amount of data

[JM_O]

Hello,

I am new to labview.
I would like to acquire some signals using an USB device.
I would like to use it's full bandwidth capability (16 bit 1MS/s shared between my channels) for 10 minutes.
During the acquisition process, I want to see all the data captured so far (not just the 10000 last samples).
At the end of the acquisition I want to be able to use cursors on the whole plotted waveforms.
How would you architecture that to have reasonable performance?

Thanks for your help
Best regards,
Jean-Marc

[Joseph_Loo]

I think you are going to revisit your thinking a bit. If you are using a standard windows Labview, there may be a 2.5 Gbyte limit associated, I could be wrong about this figure. In addition there is a 3 Gbyte limit of memory allowed on a standard 32 bit windows. Your requirements require a minimum of 1.2 Gbytes/channel. As you can see, you may be able to display 1 channel, but 2 channels is very questionable.

If you do not have enough physical memory on your machine, I suspect it will thrash a lot, i.e., run really slow.

You will need to consider other alternatives e.g., 64 bit operating system and a 64 bit Labview. Alternatively, you could break the graph to smaller pieces and view a small section at a time, e.g., 30 seconds worth of data and allow the user to scroll back and forth.

[johnsold]

Jean-Marc,

In addition to what Joseph Loo wrote You should consider that any time a graph has more data points than the display of the graph has pixels, some reduction of the data will take place in the display process. Even on the largest of monitors you are talking about hundreds of points not millions or more. LV is pretty good at this, but it can take some time especially if the input data contains far more points than the graph has pixels. You can control it yourself and get exactly what you want on the display.

It is often useful to consider the data in these ways: Data acquisition. Data storage. Data display. Data processing.

Each of these may have different requirements for speed, size, and structure. Trying to make the program one size fits all may result in a program which does none well.

I would suggest that the data acquisition be set to stream to disk. At the end of each minute start a new file. Each file is about 120 MB which is a size which can be processed, if done carefully to avoid duplication of data. Read the NI white paper on Handling Large Datasets in LabVIEW. Maybe smaller or larger files would work better for your particular situation. Have a separate display VI running in parallel with the acquisition VI. The display VI can scan though the current minute's data or bring in data from a previous file, reduce the data to fit the graph and display that. You can program it to scan through the file, to select areas for detailed examination, take averages, or whatever the user needs to see. With some careful planning you should be able to avoid most of the virtual memory or page swapping slow downs.

Lynn

[JM_O]

Thank you both for you answers.
Actually as my bandwidth is shared between channels the file size should never exceed the 1.2GB size.
But from my tests, the acquisition stops much before the 10 minutes just because of CPU/memory reasons.

I believe the solution proposed by Lynn is the way to go even if I had prefered something easier to implement for a beginner like me...

Jean-Marc

[DFGray]

Actually, creating different files should not be necessary.  You can do it in one file pretty easily.  Newer versions of LabVIEW support 64-bit file pointers, and you can use NI-HWS to get the same benefit on older versions.  I fully agree with Lynn that you will need a disk buffer.  Some things to keep in mind:

• Write the data to disk in a different loop than you take it.  You can use a queue to move data from the acquisition loop to the storage loop.  This gives you a multi-threaded application which will perform better than doing it all in one loop.  Look up producer-consumer loop examples in the LabVIEW help for how to do this.
• Use the I16 data for storage and also store the scaling factors.  This reduces the amount of data you need to read and write by a factor of 4 (2 bytes for the I16 vs. 8 bytes for a DBL).  If you use NI-HWS, there are native functions to store the scaling factors for you.
• If you are interested in speed, the three binary formats that are relatively easy to use are flat binary (native LV file I/O functions), NI-HWS (available on the driver CD), and TDMS (native to newer versions of LabVIEW).  Performance is in that order, with flat binary being the fastest for your application (note that this order may change depending upon application).
• As mentioned above, you will need to decimate your data for display.  You can find VIs to do this in the tutorial Managing Large Data Sets in LabVIEW.

Note that modern disk drives have speeds of 20MBytes/sec up and can exceed 100MBytes/sec, especially in RAID arrays.  However, with a gigabyte of data, it will take a few seconds to get all the data off the disk and decimated.  You may consider storing several predecimated traces along with the actual data so you can view the data faster.  If you only want to view and don't care about storing it, you can decimate it on-the-fly as you take it and throw away the original, keeping only the decimated data.  Of course, then you can't zoom very well.

Let us know if you need more help.

[F._Schubert]

Hi DFGray

I'm not sure if it's possible to stream that amount of data through USB. At similar sized DAQ using PCI, it was necessary to use Interrupts in addition to DMA.

and JM_O

the problems you face here are not related to being new to LV. If you just want to play around with DAQ to learn LV and DAQ, have fun exploring these limits. If you have some specs to fullfil, plenty of Forum users will help you to find a workaround...

Felix

[DFGray]

I checked with our driver gurus, and you should be able to stream 1.25MS/s over the USB port using that DAQ board.  If you can't, we want to know about it.