Read time of tdm file increases

[jsab500]

Hello,

I recently recorded a lot of data in tdm format. I originally saved it in an excessively organised and complex way which resulted in large header files and was very slow. I changed it and most of the data is fine, except 6 files in the old format. Just trying to read them to get the data out and then convert it into an easier format is taking far too long. Each file has ~15000 channel groups, each with 3 channels (I won't do this again...). I have witten a small vi to test the reading speed. Excluding opening and closing of the file, if I just find a channel group by name, reading one channel from that group takes about 1 second. This would still take days to read it all but would be acceptable. The strange thing is, is that if I do this in a loop for successive channel groups which is clearly necessary, it takes longer and longer each iteration. This is behaviour that I have always seen when reading tdm files, but it hasn't caused a serious problem before. I can't find any information about why though, so I hope I'm just doing something stupid. I have attached the speed testing vi (Labview 2010). Running it for 5 iterations, the time taken to read the channels is

0.82304 s

1.58809 s

2.42514 s

3.56820 s

5.60632 s

The channels it is reading all have the same number of values, and it makes no difference if I start at a different channel group (by adding a number to i in the loop).

Does anyone have any explanation for this behaviour?

Thank you very much for your help, and I'm sorry if it's something tht has been asked before,

James 

[niNickC]

Hello,

When looking at this, my first instinct is that this is a memory issue. I am the moment I am unsure of what exactly could cause these problems but could you verify if this is the case? (I assume you are using windows) Could you open the task manager whilst running your application and observe the memory and CPU usage. Also can you run your VI for a few more iterations, maybe run it for 10 minutes so we can see if the increase is linear or follows some other pattern?

Best Regards,

[niNickC · Accepted solution]

Hello again,

Having another look at your code, I think I have a couple of suggestions as well.

Try changing the overwrite options in the open data vi to open (read only).

Also, I found one instant of somebody else having this problem. Their solution was to use the "Convert to TDM to TDMS VI" in the storage palette to convert the TDM files to TDMS and then doing the read operations. Could this work for you?

Best Regards

[jsab500]

Thank you very much for these responses. I will try both things and report the results when I have time.

[jsab500]

Ok, I changed the option to Open (read only), and it was taking a consistent 3-5 seconds to read each channel. As this was slower than before, and there were a few other things running on the computer, I restarted it and ran just this program. It then reverted to the behaviour mentioned above. I had changed nothing, it still was read only. Weird.

Anyway I converted it to TDMS first as you suggested, which is working. Looks like it will take ~24 hours to process it all, instead of ~a year.... So thanks! The TDM to TDMS converter must read the TDM to do the conversion musn't it though? So how does it do it so mcuh faster than the provided read data function?

Thanks very much

James 

[niNickC]

TDMS is an all binary format whereas TDM posses an XML header, otherwise they are very similar. Being a binary format you would expect it to read faster especially as you mentioned that the header was complex.

There are a number of un-closed references in your code and I wonder if this was partly the issue, although this wouldn't explain why converting to TDMS fixed the problem?

Beyond that, it would be nice to try this without using the express VIs to see if the problem still occurs. 

Also, where you able to see if this was a memory issue? And if the read time continued to increase beyond what you posted?

Anyway, I am glad we found a solution to your problem.