What is the most efficient way to store 20KHz of data for a 24hour period?

[AustinPeter]

Hi,

My labview program needs to store 4 channels at a speed from 20Khz up to 50Khz of data from 4 analogue input channels (usb daq)  at the end of each day. and continue this process for up to a year,

was just wandering what the best way to store this would be, as txt files are proving to be up to 100GB per day.

I am converting 3 double precision numbers data to txt format and writing to a txt file currently.

I am sending double precision data to txt format and writing to a txt file currently. I need to be able to analyse this data at the end of each day then send it to an online server.

really appreciate any help.

thankyou.

[johnsold]

Raw binary is considerably more compact than double -> text.

However, 4 channels at 50 kS/s and 16 bits/sample still reaches >34 GB/day. or 1.26e13 bytes/year

Can you do some data processing before you store the data? What kind of analyses will you be doing before sending the data to the server? Perhaps some of that could be done while collectng the data.

A white paper on handling large data sets exists on NI's web site. I do not have a link at the moment, but you should be able to find it without too much trouble. It will have some useful tips on how to handle the data and how to avoid unecessary copies in memory.

Lynn

[smercurio_fc]

Can you work with single-precision numbers? That would cut your file size in half.

I would also suggest compression. LabVIEW has compression functions. However, curiously enough, if you incorporate compression, it turns out that saving it as text may end up being better in terms of file size. As an example, I created a VI to generate an array of 1 million DBL values and saved it to binary. The file size was 7.6MB. Compressed it was 7.3MB. When the 1 million values were saved as text the file size was 9.7MB, but compressed it was 3.5MB.

[patrickthebold]

That's really weird about the compression.  Was the data equally random in both cases? 

[altenbach]

---

@smercurio_fc wrote:
However, curiously enough, if you incorporate compression, it turns out that saving it as text may end up being better in terms of file size.

---

This probably depends a lot on the format you are using. Formatted numbers typically have less information that contained in the original binary so I would expect the compressed size to be smaller. 😉  From your numbers, it seems that your text format takes a biit over 10bytes/number, not even enough to retain the ~15 decimal digit resulution of the DBL. You even waste ~1MB on sperators and decimal points each. If you use scientific format, there won't be much left for the mantissa. 😉 Your compressed text file size is smaller because it contains significantly less data to begin with. 🙂

Back to the original question:
So, what kind of data is this? For example if this is audio data, maybe a lossy compression such as mp3 would be acceptable?
What is the resolution of your DAQ system (12bit, 16bit)? You definitely don't ned to waste more bits/sample than that in the file?

[AustinPeter]

Thankyou for the responses you are certainly heading in teh right direction.

basically I need the values to 3dp. and they never go over 1000. ie the biggest number i will ever need is say 1000.999

I dont need to analyse the data in real time, just need to be able to access the file at a later date and unpack the data to do the analysis on,

storage size really is the key as at the end of each day will be transferring the data accross a network and will be paying per megabyte.

i need to store four channels at a time. i was thinking that binary format was the best way, but like i said have been using txt files. any more ideas? and examples and links to white papers would be really useful. thankyou.

[altenbach]

---

@nistudent wrote:

basically I need the values to 3dp. and they never go over 1000. ie the biggest number i will ever need is say 1000.999

---

Are the numbers always positive? what is the smallest number? We need to know the range!

The best format will differ if a smallest number is -10000.999 or 10000.100, right?

So we know you only need a step size of 0.001. How many different values can your data have? 

[AustinPeter]

sorry,

the range is -1000.999 - 1000.000 as is a sinusoidal signal.

the resolution is 0.001, do you need anything else?

[Herbert Engels]

The file format NI recommends for your kind of use case is TDMS. It can be written and read with the TDM Streaming functions, located as a sub-palette in File I/O. TDMS can easily handle the data throughput and the amount of data you are asking for, even if all 4 channels are acquired with 50kHz. The resulting file for 24 hours (given 4x50kHz with single precision floating point numbers) would be about 70 GB. There are several examples that show how to handle TDMS files, plus LabVIEW has a viewer for TDMS. TDMS is also supported in several other NI and 3rd party applications.

If you want to reduce the disk footprint, you could write unscaled data to disk. That means, if the unscaled data you acquire is a 16bit integer, you would cut the disc footprint in half. The TDMS examples include VIs that demonstrate how to write and read unscaled data in TDMS.

Hope that helps,
Herbert

[johnsold]

To get 0.001 resolution on a range of +/-1000 requires 21 bit resolution in binary (2^21=2097152) and nearly optimal mapping of the signal to the converter range. Does your A/D converter have >= 21 bit resolution? Most 50 kS/s converters do not. You cannot create resolution on a bit by bit basis beyond what your hardware will produce, so saving data at higher resolution is wasting bits.

If your data is a sinusoidal signal you need much less data to represent it. Three numbers, amplitude, frequency, and phase, tell you everything you need to know about a sine wave. If you simply record those numbers when they change along with a timestamp, you will have much smaller files.

Lynn