What is the most efficient way to store 20KHz of data for a 24hour period?

[AustinPeter]

thankyou the tdm stuff looks most interesting.

the sin wave is not a standard sin wave, i was just using that to give you an example of the type of data we were getting. it is a random which has a sin wave undertone to it. I take the point that it might well be easier to transfer the number to a value which is always a plus number and then output that in labview to a file instead of keeping the range.

again if anyone knows of any white papers on this then it would be most appreciated, as have to handle the data storage and reloading of the data to analyse at a further date.

thankyou

[Herbert Engels]

Try this one: http://zone.ni.com/devzone/cda/tut/p/id/3539.
And, if you have to know, this one: http://zone.ni.com/devzone/cda/tut/p/id/5696.

Herbert

[smercurio_fc]

Well, there has certainly been a fair bit of discussion on this post since my initial post and your question. To answer your question (though it probably doesn't matter now), the data was random both times. I was simply using the random number primitive. To get the text file I was simply using the Number to Fractional String function. My example was more of a demonstration to indicate that while compression may be an option, the results will be different based on the original data. The "curiosity" comment was more to point that out. I wasn't the one who was curious about it since I know that the results are going to be highly dependent on how the data is formatted in the first place. For instance, with the example I put together I had 6 decimal places for the random numbers. This is usually more than enough for most applications.

As I said, the discussion has moved on from there, but I'm just responding to your query about the data that I had used.

[PhillipBrooks]

Here's an idea. I worked with an instrument called the dewk 1620 from Fluke. It uses a binary format to store data to internal memory. The file can be transfered via RS232 fairly quickly, but needed to be parsed using a tool called LoggerWare. I contacted Fluke to get the format of the file for direct parsing from LabVIEW, but never sucessfully managed to parse it completely.

The trick was that the logger recorded deltas until either the delta rolled over or a period expired (I think one hour). You could define your own data types in your file format such as "REFERENCE", DELTA", etc.

If your values change slowly, you could record incremental changes with as little as six bytes/record; one byte for the type (REFERENCE, DELTA) and 10 bits per channel ( .001 resoution ) times 4 channels for a total of 40 bits or 5 bytes. If your data changes by say 1.0 units/second, you'll need to add a lot of REFERENCE records and this won't help much.

If you have slow changing data, you could get down to maybe 11-12 GB a day.

Again, just an idea...

Message Edited by Phillip Brooks on 07-16-2007 01:36 PM

[DFGray]

A couple of tips.  If you use TDMS, make sure you use TDMS, not TDM.  TDMS was optimized for streaming and has much less overhead than TDM.  If you want/need compression or 2D array support, try NI-HWS.  NI-HWS is based on HDF5, so can be read by most analysis packages (Matlab, Mathematica, etc.). 

[Otman]

Phillip,

I'm working with the same instrument. Did you try communicating over TCP/IP? We're having problem with the LogWare software and we're investigating the possibility of managing these dewks ourselves through a LV program.

Any tips on your experience working with these instruments over TCP/IP is greatly apprecaited.

Otman Estrada