Large file size of TDMS files

[llanger]

Hello everybody,

We are currently generating Data with a file size around 600 Mbyte. This is done on a Windows based test rig with lab view installed. My job is to generate Reports in Diadem (that’s the reason why I am posting it in diadem).
At the moment I have big problems to handle the data – Diadem is very slow and sometimes it crashes after a couple of minutes.
The data files contain around 140 channels with 300k data points each.
May be it is normal, that diadem is getting slow, but when I save the data to CSV, it is getting half the size – around 300 Mbyte. If I am using 7zip or similar, the file size is shrinking to 1/10^th of the size.
I was trying to defrag the file in lab view, it did not work at all. File size after defragmentation was almost the same. Are we doing something wrong here?
Thank you in advance for your help!

Lars

[AndreasK]

> I was trying to defrag the file in lab view, it did not work at all.

What does that mean? It crashed or never finished?

If your tdms file has 600MB how big is the tdms_index file after opening in DIAdem.

From start I would suggest that it is a very heavy fragmented file which can be  pooved by a big tdms_index file.

Could you provide a small example file. (Same kind of content but not that big?) 

[llanger]

Hello again,

I will attach a example file including a index file. The data is similar to the big file, but the logging time is way shorter.

The defragmentation was working till a new file was generated, but it had the same size as before.

The file size of the index file is around 4... 38 MB

[AndreasK]

I checked with your file and calling defrag changed the behavior of DIAdem drastically.

Loading your small example:

Original: 11 seconds

Defragmented: 0.6 seconds

Why din't you see the performance increase?

When DIAdem loads a tdms file it only reads the meta information.

The channel data is read when needed.

So if you want real info you need you have to specify that load should happen on open.

option explicit

data.Root.Clear
LoadFile "orig", currentscriptpath & "orig\29012020_1535_PHP-36565_Release_6_Mono_LT_No_NoiseRed.tdms"
data.Root.Clear
LoadFile "defrag", currentscriptPath & "defrag\29012020_1535_PHP-36565_Release_6_Mono_LT_No_NoiseRed.tdms"

sub LoadFile(byVal info, byval filePath)
  stopwatchreset 1
  Dim ip
  Set ip = Navigator.Settings.CreateImportParameter("Load")
  ip.BulkDataLoadingMode = eBulkDataLoadingImmediately 
  DataFileLoad filePath, "TDMS", ip
  LogFileWrite info & ": " & stopwatch(1) & " s"
end sub

shows

orig: 11,2977693 s
defrag: 0,6466449 s

If you do not want to defrag the files you can set load behavior to "Always load bulk data".

DIAdem also contains a command to defrag TDMS files:

TdmsFileDefrag

which defragments the file inplace.

The help shows how you can use DIAdem Datafinder to find files that need defragmentation.

---

How are the files measured in LabVIEW?

[llanger]

Hmmm, may be I did something wrong. I will check all these things you described and will let you know whether it worked.

Thank you a lot for the nice description.

[llanger]

Fine, it is working.

Diadem works stable now and the reaction time in VIEW and REPORT is way faster. Seems that the files were highly fragmented.

Thank you very much!