Large rate of data acquisition

[TomPurves]

Hello all,

I am trying to take samples using the express VI, at a rate of 250ks per second for 15 minutes. I am having trouble pushing it to any longer than 3 minutes so I feel like there is some memory management that I'm not fully understanding. I run task manager alongside the acquisition and I seem to have physical memory available (8GB RAM). I have tried running the VI in both continuous and N samples mode but nothing seems to work. The cont mode seems to run for longer but it fails at the 3-4 minute mark.

Have I missed something?

Best

Tom

[RavensFan]

Need more info.

What do you mean by it "fails" at the 3-4 minute mark?  Do you get an error message?  Are you logging to a file and it stops saving data?  LabVIEW crashes?  Your PC crashes?

Can you post your VI so we can see how you have the express VI set up?

For that high of a data rate and length of time, you may want to get away from express VI's and learn to use the actual DAQmx functions.

[TomPurves]

Hi there,

I was intending to save to file after the acquisition has completed, I've attached my VI for reference.
The error may be something to do with appending the waveforms perhaps, but I'm not sure how to take continuous data without appending. The error message is 'not enough memory to complete this operation' followed by 'error - 200279... the application is not able to keep up with the hardware acquisition' 

Many Thanks

Tom 

[crossrulz]

It should also be stated that 32-bit applications are still limited to the 2GB of memory.  What are you doing with the data?  How many channels?

As said already, please post your VI and that will answer most of our questions.

[RavensFan]

You still haven't said what the error is.

I see you edited your message since I started typing and it confirms what I typed below.

I'm going to assume it is some sort of out of memory error.  Let's do some math:

250,000 samples/sec * 60 sec/min * 15 min * 8 bytes/per sample = 1.8 billion bytes,  or approx. 1.8 GB.  That is a huge amount of data.  And since you are continually appending data in array (hidden in the express VI)  when the array fills up, LabVIEW needs to scramble to get Windows to find another larger chunk of contiguous memory to move the data to.

Don't keep appending data to an array.  Look up Producer/Consumer.  Send the data to a parallel while loop using a queue so that the data can be written out to file as more is being collected.

[crossrulz]

---

@TomPurves wrote:

I was intending to save to file after the acquisition has completed,

---

No, you do not want to do that.  You should be saving the data as it is acquired.  If you look in the DAQ Assistant, you will see a tab for "Logging".  Enable the TDMS logging and set it up the way you want.  This will eliminate the need for a constantly growing waveform and will save the data very quickly to a TDMS file.

[TomPurves]

Thank you for for the help - suggested below is writing to a TDMS file. Do you think the will be suitable for the large amount of data I would like to take?

[TomPurves]

Thanks - as an aside, what is the best way to extract just the voltage values that I'm reading to a .txt file.

Thanks again,

Tom

[RavensFan]

As I said, a producer/consumer using queues to send to another loop.  That consumer loop can convert the voltage data to text and write it to the file.

[crossrulz]

---

@TomPurves wrote:

Thank you for for the help - suggested below is writing to a TDMS file. Do you think the will be suitable for the large amount of data I would like to take?

---

The DAQmx TDMS Streaming (which is what the DAQ Assistant is using) was designed specifically for situations like this.  If you don't mind a binary format (that can be opened in Excel with the plugin that I am sure you installed with LabVIEW), then I would do this.  If you really want a text file, then you definitely need to go with a Producer/Consumer to log the data as it is being read.