How do you handle and distribute huge amounts of data?

[Frank_Rizzo]

I just downloaded the evaluation version.  I'll let you know or you could do the same.  I think the thing about DIADEM is that its for doing the analysis yourself....im looking to distribute the data to people who prob wont have NI software......

[Sima]

Yes, but I meant in terms of TDM(S) files. In terms of file size, would it be better to go TDMS, simple 2D dbl array binary file, text file (bad), excel (worse), etc?

Hey, I can take this to the DIAdem forum 😉

Message Edited by Sima on 08-01-2007 12:28 PM

[Kevin_Price]

I'd recommend TDMS.  The files may be slightly larger than regular TDM but are much more efficient to create at run-time.  It's also more flexible and self-descriptive than a raw binary write.

I haven't used Diadem, but strongly suspect it provides ways to convert / export your TDMs data into other forms such as ASCII or whatever.  I would think that a reasonable overall approach might be for you to use Diadem.  You'd use LabVIEW to create TDMS files, then bring up Diadem to export the data into an agreed-upon format your researchers can handle with their analysis package(s) of choice.  They would never need to see the TDMS files, nor would they need to have LabVIEW or Diadem.

-Kevin P.

[TCPlomp]

So what they want to see 400 meg of data, give 't to them.

Use the TDMS viewer of NI and transform it into an extraction feature where they can select channels and do some basic filtering.
The tool could extract parts of the data to shiver shiver a csv file.

Or just shove DIADem into their faces.

Ton

[BillMe]

I think that using TDM/TDMS file format and DIAdem as the data mining and reporting tool would work well as a turnkey solution. DIAdem has no such limitations as Excel, in fact, the amount of data you can read in to DIAdem is limited only by available memory on your PC.

Download the free 30 day trial here.

Message Edited by Bill@NGC on 08-02-2007 12:01 AM

[Ulrich_Bierwisch]

Hi,

TDM/TDMS is actually exactly doing what Yuri33 proposed. The raw data is stored together with textual information about channels , calibration and lots of details about the test.

DIAdem is also the best way to handle large amounts of data. 312.000 values in 315 channels is large but not unusual in DIAdem environments. The data, if stored as 2 byte integers, should take 200 Mbytes on Disk and less than a Gig as fully calibrated doubles. I would expect a reasonable reduced filesize if the data is ziped for transportation (assuming that the values are not white noise).

Of course it might be a good idea to reduce data before sending. But in many cases, people like to have the oportunity to zoom into details and repeat new ways of analysing with old raw data. A dataset like this is worth a lot. Today, you can save 1000 sets of this data on an external harddisc at the cost of 100$. There is no need to throw away precious data. The TDM-Header is able to hold lots of additional information about each measurment and the DIAdem-DataFinder it able to find the data you are looking for.

Ulrich Bierwisch

[cs42]

I always read about TDM/TDMS for such solutions. Why not use the LabVIEW Datalogging and Supervisory Control module? Saves data to a database which can easily be accessed using Diadem. The only thing I don't know, how it works with such sampling rates.

[Frank_Rizzo]

Ulrich,

Thanks for the info.  My problem is I am looking to distribute the data to people who will not have any NI software.  Is Diadem able to convert the entire TDM/TDMS files into something I can easily distribute?

[Ulrich_Bierwisch]

Frank,

The SaveAs-Function in DIAdem is supporting several formats to save the data. The problem is, that there is no all purpose standard for binary data. In most cases, some kind of simple ASCII Format is used for this kind of distribution. Usually, a simple format is used to be imported in most softwares. Your data is just a block of channels of the same length which makes things much easyer.

Using ASCII is not as bad as you might think. The filesize reduces a lot if it is ziped.

DIAdem has several formats for this:

Textfile *.csv is a very simple ASCII-Format which can be loaded in Excel.

LVM files *.lvm is a simple ASCII-Format for LabVIEW-Express VIs. You can write and read this filetype with LabVIEW and DIAdem. It holds some additional information about the data. The format is designed to be human and Excel readable.

Excel *.xls might be OK for all Excel users, but your channels are probably to long for most versions of Excel.

More filetypes are available as dataplugin. Look at www.ni.com/dataplugins for the latest list of plugins. Unfortunately, most of the plugins are for reading only. It is easy to create dataplugins. If your users are prefering a special format, it is posible to write a plugin just for this purpose.

Ulrich Bierwisch

Message Edited by Ulrich Bierwisch on 08-03-2007 09:28 AM

[DFGray]

I must politely disagree with my esteemed colleague.  There is a widely used and supported binary file format called HDF5.  Most analysis packages (e.g. Mathematica, Matlab) have read filters, if not write filters.  It is self-describing and hierarchical (HDF5 is short for hierarchical data file version 5).  It has native compression.  It is supported in LabVIEW in two ways.  First, NI-HWS uses HDF5 under the hood.  This file toolkit is available on the driver CD next to the computer based instruments (which use HWS as their default data storage format).  The second is to call the HDF5 shared code directly from LabVIEW.  An old toolkit, called sfpFile (an early version of NI-HWS) can be found here.  Ignore the top level files (which implement NI-HWS) and use the HDF5 files, which expose the low-level HDF5 interface.  Be aware that the learning curve is steep and the HDF5 DLL is NOT thread safe (make sure you do not try to call into the HDF5 DLL from two places at once).  You can search this forum for HDF5 for more info and/or go directly to the HDF home page.

Streaming to disk with HDF5 from LabVIEW is essentially hardware limited, with one important caveat.  HDF5 stores indexing information in a B-tree.  If you are streaming large numbers of objects/channels (you seem to be), the B-trees are balanced more often than streaming a single channel.  This results in periodic delays in stream speed.  However, since your average speed is about 1MByte/sec, more than an order of magnitude under what HDF5 can do, you should not see any problems.  If you do, a simple double buffer will cure them (you should be using a producer/consumer architecture anyway, which automatically does this for you).  Note that TDMS does not have this particular problem.

If you decide to go the NI-HWS route, let us know so we can tell you how the data is formatted.  It is essentially a 64-bit version of SCPI DIF with a lot of enhancements.