How to improve TDM open speed

[Jim West]

I have been trying to improve the data loading performance of my application that uses TDM files.  Overall the file loading speed for an example file is about 400 ms.  Some of the data sets that my application manipulates are 500 files in size.  This leads to a 3 minute load time.  In my investigation, I found that the Labview TDM “Open Data Storage” node represented about 300ms of each 400ms.  So I thought I would be clever and wrote a routine that spawned off processes to open each file ‘in parallel.’  I wanted to continue opening files while I was loading data for another file.  Unfortunately, it doesn’t run any faster.

I wrote a test program attached to see what kind of parallelism could be achieved.  NONE - The Serial run time is the same as the parallel run time.  

I would expect to see some improvement in speed for the parallel version but there must be a non-reentrant component that is serializing everything.

Does anyone have any ideas on how to improve the speed of opening and loading the dataset?

My test program and sample data is attached.

Jim West

Summitek Instruments

[Brad_Turpin]

Hi Jim,

Neither the Storage VIs nor the TDMS VIs are re-entrant in LabVIEW 8.2.  Speed testing at NI indicates that the TDMS files open in LabVIEW much faster than the TDM files, so if possible switching to TDMS may remove the speed problems you are seeing.  R&D has an interest in making these VIs re-entrant, but there's just no telling at this point when or if that will happen in the future.

Regards,
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[Jim West]

I wanted to finish this thread with some real numbers.

I converted the files referenced in a prior post to TDMS then converted my routine from the TDM to TDMS library.  There was significant improvement in loading speed.  Went from 400ms to 16ms for the same file.  (only in 8.2 though users of prior versions of LV are out of luck)

I'll point out that the TDMS library is not a clean drop-in replacement for the TDM express vis.  There was a little bit of work to extract all of the properties I was using.  The properties are in the drop-downs in the TDM while TDMS requires a specific calls to get the properties.  I got it working pretty easily but I was surprised that the FILE properties showed up as another group with the name of the file as the group name.

The TDMS file browser is neat and shows how to do almost everything.  Check it out first.

Jim West