Periodic spectra storing in TDMS

[bilik]

I need to periodically store spectra from 4 voltages and 4 currents, each  spectra signal has 100 spectral lines. This means 800 spectral lines is one record which is created each aggregation interval (200ms up to 10 min – user selectable). For long time I use binary data format for this and it works fine and fast. Now I need to optionally store data into TDMS data format for my customer who want to process them in DIAdem.

I found only one example for periodical storing of spectra signal: Express write data (time and frequency domain).vi. I tried to create data file with approximatelly 500 records. The data file is about 2MB. But to show this file in LabVIEW TDMS File Viewer takes tens of second having CPU utilisation on 100%!!! If  I load this 2MB file into DIAdem it makes also significantly longer responses in comparison with example files!!!

I am looking for efective way how to proceed for this type of data with TDMS format. Consequently I will need to read data in LabVIEW, but the reader must be fast. I have few typical tasks for reader:

• To read time sequence of selected harmonic on selected signal for the whole measurement
• To read one record (all harmonics) from one signal

Please help.

Bilik

[AdnanZ]

Since I am not sure how you have programmed and setup the whole thing, I won't be able to suggest anything with regards to designing. One thing I would suggest is to try running your file through the TDMS Defragment node in LabVIEW. You should be able to find the node in the File I/O -> TDMS palette.

[bilik]

Hello Adnan,

I have used the LabVIEW 8.6 example, no programming from my side. Please find attached  TDMS - Express write data (time and frequency domain).vi

I want to store spectra data into TDMS (as  described above) in LV application providing gap free DAQ process and making extensive calculation - there is no time to do TDMS defragmenting on-line. Anyway at this moment I am struggling with effective and fast TDMS data file read or processing (spectra data stored by attached example). By my opinion the way how example store spectra datais not the best. I have long time experience wit LabVIEW, but very little experience with TDMS. I am pretty sure, that periodical storing of spectra data is nothing special and there must exist effective solution.

Thank you in advance for help.

 bilik

[AdnanZ]

I just run the attached example on my computer for 800 iterations and it opened the TDMS file using the File Viewer in a matter of seconds. Can you post some of your benchmarks?

[Brad_Turpin]

Hi Bilik,

I followed your link from your DIAdem forum post to here.  I'm confident that we can get your TDMS streaming to work quickly in both the LabVIEW TDMS viewer and in DIAdem.  But if you would really prefer to stick with your existing binary file, another option would be for you to create a DataPlugin that reads your existing binary file.  You can give this DataPlugin to your customer who wants to process the files in DIAdem, and then DIAdem will treat your binary file the same way it does TDMS.

You can create DataPlugins using both VBScript and LabVIEW-- programmer's guides are listed at www.ni.com/dataplugins.

Brad Turpin
DIAdem Product Support Engineer
National Instruments

[Herbert Engels]

You're probably creating a very large number of groups and channels in your file. Since you are storing data at regular time intervals, you might not need to do that, you could just keep adding data to the same channels every 200ms.

Hope that helps,

Herbert 

[bilik]

According to long open time for data file from my example: after my laptop restart it tooks few seconds in DIAdem and about second in LabVIEW TDMS FileViewer. Sorry for this, no idea what was wrong with reading TDMS. Anyway I am struggling with duration of TDMS write and correct structure of TDMS file for my application.

My idea is to switch from binary data to TDMS in my application: Power Quality Analyzer firmware

http://www.elcom.cz/en/virtual-instrumentation/products/ena/clanek/393-software-for-power-quality-mo...

I prepared two vi demonstrating storing same data into BIN and TDMS file. The difference for same amount of data is huge:

-BIN file is writen and closed in a moment.

-TDMS is written in about 8 seconds, closed in 2 seconds and eventually defragmented again in few seconds.

My questions are:

- can be found more effective structure of TDMS file

- is TDMS file usable for periodic on-line data storing of spectra data in parallel with extensive calculation?

Of course the easiest way is to stay with binary format and make dataplugin, but I want to use one TDMS file for periodic data and event data - this seems to me like great feature. Now I need to use several files with different suffixes for one measurement (e.g. 1 day, 1 week).

I will appreciate very much any suggestions.

Bilik

[bilik]

Could somebody help me  with described topic?

Thank you very much

Bilik

[Brad_Turpin]

Hi Bilik,

One reason you are getting poor performance in the TDMS code you posted is that you are writing one data point at a time to each channel.  Since each buffer write in TDMS is prefaced with a binary buffer header describing its contents, you are creating a binary header for every data point you are storing.  The LabVIEW 2009 Beta, which is already live, contains TDMS VIs with an updated option to create a single binary header for the entire file (and forego the option of changing anything during the acquisition)-- with this option you would not encounter this problem.

But you can sidestep this problem with already shipping LabVIEW versions by invoking a special channel-level property set to automatically buffer channel writes up to a certain minimum buffer size, then flush that minimum buffer to disk automatically.  This avoids excessive binary header writes without changing your code inside the acquisition loop, though it does require you to add the channel property sets outside (left of) the acquisition loop.  Just assign the "NI_MinimumBufferSize" property to something like 500 or 1000 or 2000 values on each of the channels.  This also avoids the need to defrag the TDMS file afterwards, since you aren't creating it with serious fragmentation to start with.  However, I did notice after adding all this to your code that the write performance of your binary file is still much better than TDMS, and this I don't understand.

Perhaps Herbert can take a look at the edits I made and suggest further improvements.

Brad Turpin
DIAdem Product Support Engineer
National Instruments

[bilik]

Brad,

thank you for your effort. I tried examples modified by you and it seems to me storing into TDMS is too slow and extremelly CPU consuming to be used in gap-free DAQ process and analysis.  Have a look on CPU utilisation for TDMS example - continuous 100% on my laptop. If somebody will find solution how to proceed better I will be really happy.

Anyway I still do not know what is the best TDMS data structure (even for BIN to TDMS converter) for spectra data I described. I need to effectivelly process and display data in DIAdem in two ways:

-instant spectra for selected time(one record of all harmonic)

-level of selected harmonic in time interval (all record of selected harmonic)

Thank you in advance for any help

Bilik