Write Data storage VI performance issue

[Elizabeth_S]

Hello,

I have been working with your VI and have not been able to reproduce the same problem that you were describing.  You mentioned that the for loop containing the Write Data.vi is slowing down over time.  Your main for loop is only executing one time, so I am not sure how you determined the slow down. 

1)      How are you determining the slow down?

2)      Which loop are you referring to?

3)      How many iterations does the for loop make?

4)      Do you mean each time you execute the program?  

Could you post your larger bin file on the National Instruments FTP site? 

ftp://ftp.ni.com/incoming

If so, please upload a zip file and label it with your username.  Please post back with answers to the questions above and let me know if you are able to upload your bin file.

Regards,

Elizabeth S.

Applications Engineer

National Instruments

[Seaman008]

Elizabeth,

I decided not to pursue this problem any further.  Instead, I made a new VI to write the tdm/tdk file pair with my own code and without the storage VI.  The new VI is at least an order of magnitude faster so I'm satisfied with it.

S.

[Jarrod_S.]

Seaman008,

If you get a chance to upgrade to LV8.2, I would strongly encourage you to look into the new TDMS VIs that balance the flexibility and organization of the TDM format with the ability to stream data to disk at very high rates that do not suffer from the performance slowdown you have been seeing with the storage VIs.

The problem is that the basic TDM file format is optimized for reading data from disk for extremely fast analysis and number-crunching capabilities. It was not designed for streaming data. The issue here is that data in a TDM channel must be contiguous in the file. So if you want to append data to a channel in the middle of the file, the Storage VIs must first push everything after that channel back to make room for the new data. As you have seen, this causes an exponential slowdown.

TDMS does not have this limitation. It uses special indexing to allow one channel's data to be spread throughout the file. That means you can append data without affecting any other channel data in the file. So streaming speed does not decrease as your file gets larger. This does mean that TDMS is not as quick when reading data back from file, because it first has to collect all the relevant data for a channel spread throughout the file. There is, however, a defrag function you can call offline to put channel data in contiguous chunks for very quick read access.

I'm glad you found a solution for your issue. Another solution that would allow you to keep using the Storage VIs would be to close your file and open a new one every so often, so that the file size never gets too large. Then you could have a tool that would merge all your files offline into one large file.