save a big data from DAQ

[labmaster]

I have a trouble in saving a big data from DAQ in realtime.

I want to save a high rate data of 20 MS/s using PXIE-5172, oscilloscope in PXIE chasis but got the trouble of "memory full".

I generated a random number of 20M*3channels "SGL" array.

Using TDMS or Save to Binary file format, it takes 2.8 sec in each SGL array (~9 sec for DBL array).

If so, the data cannot be saved continuously in realtime even though I remove all required routines for display and analysis in a separated consumer loop.

Is there a known technique for this in LabVIEW or specialized hardware except PXIE chasis?

Do you have some idea to overcome for fast acquisition and to handle its data ?

labmaster.

[GerdW]

Hi labmaster,

---

@labmaster wrote:

I have a trouble in saving a big data from DAQ in realtime.

I want to save a high rate data of 20 MS/s using PXIE-5172, oscilloscope in PXIE chasis but got the trouble of "memory full".

I generated a random number of 20M*3channels "SGL" array.

Using TDMS or Save to Binary file format, it takes 2.8 sec in each SGL array (~9 sec for DBL array).

If so, the data cannot be saved continuously in realtime even though I remove all required routines for display and analysis in a separated consumer loop.

Is there a known technique for this in LabVIEW or specialized hardware except PXIE chasis?

---

Simple math: handling 20MS/s for 3 channels using SGL datatype requires to save 20M*3*4=240MB/s! (For DBL data it's 480MB/s…)

You need to have a really fast harddrive to achieve such write rates. Do you use such SSDs?

[labmaster]

Thank you for your reply of my silly question.

I don't think PXIE-8840 controller have the high speed of HDD.

It contained a 5400 rpm HDD corresponding to 100 MB/s, theoretically.

As I expected, is it impossible to save whole data for fast aquisition?

Or do I need to replace HDD with SSD (520 MBps)?

I am wondering if how to use the NI product of high rate DAQ in practical application using PXIE-5172, oscilloscope.

Usually, just save the result of analysis with the raw data in this application?

Any comment should be welcomed.

labmaster.

[Bob_Schor]

As so many Forum poster do, you failed to attach your code, so we can't judge if there may be ways of getting better throughput, and you haven't explained well enough your underlying problem (how much data are you acquiring, including how many channels, at what sampling rate, and for how long per continuous sample).  Does the "domain of your problem" require terabytes of data to be collected "all at once" at near GHz rates?  You might need to wait a few years for technology to catch up.  Can the problem be broken down into "reasonable" segments?  Maybe then we can make useful suggestions.

But in order to get useful answers, you need to ask clearer questions and let us see both (a) what you are doing (in VIs, actually code we can open and inspect, even run), and (b) what you (ideally) want to do (in clear, descriptive words, not necessarily in VIs).

Bob Schor