DIAdem taking a long time to load 700MB TDMS file to data portal

[yin_the_novice]

I'm trying to load a 700MB TDMS file into DIAdem ,4 channels 10Hz sampling rate over 4 days . When i drag the file into the data portal , diadem freezes up , its on a 2GB machine, file located on the desktop, not running any other programs and it went 50min without being able to load before i made it quit. seems to work fine on smaller files though . Was wondering if this sounds normal and if anyone knows a way around it.

yin

[YongqingYe]

Hi yin_the_novice,

Do you have LabVIEW installed on your machine? Maybe you can try in LabVIEW to use "TDMS Defragement" to defragement the TDMS files firstly and then try again in LabVIEW to see whether there's any improvements. 

"TDMS Defragement" is in Programming -> File I/O -> TDM Streaming.

[Otmar]

Hello Yin,

Have you tried "Register loading" the file? When you find the file in the directory, right click and chose "Register Data.

That speeds up the loading process by using the actual data file on the hard drive rather than transferring the data to DIAdem memory.

    Otmar

[StefanR]

Hello Yin,

does your TDMS file contain explicit time channels?

[Brad_Turpin]

Hi Yin,

How big is the corresponding *.tdms_index file?  If it's a coupel kB, then you don't have to worry about defragmenting.  If it's 10% or more of the size of the *.tdms file, then you are seriously fragmented and should consider defragmenting.

Does the file open in the TDMS file viewer?

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[yin_the_novice]

Thanks for all your input. 

YongqinYe and Brad : I've tried defragmenting but this also seems to be taking a  long time (in excess of ) on 2 different machines. The index file is 300MB, so about 75% of the *.tdms, 

Otmar:  i also tried registry loading the data but this still took a long time  on this particular file although it worked great on smaller files (5MB). 

Brad : the file does not open in the labVIEW TDMS viewer  although i'm able to view it in chunks of 65000 using the TDMS importer for excel. 

Stefan: the file does have explicit time stamp channel . 

Any extra help would be appreciated but the way I'm thinking of doing it now is to use much longer buffer size (from only 6 to a few hundred) when collecting the data to decrease the level of fragmentation. 

[StefanR · Accepted solution]

Hi Yin,

you can defrag the TDMS file while loading into DIAdem programtically:

Dim TDMSFileName : TDMSFileName = "C:\Dokumente und Einstellungen\GrRomaiS\Eigene Dateien\LabVIEW Data\ttt.tdms"
Call DataFileLoad("<filename>" & TDMSFileName & "</filename><defragbeforeopen>yes</defragbeforeopen>","TDMS","")

Please note that in this case you need to replace at least "<" and "&" in your filename (not contained in example code above).

Additionally we incorporated a fix into DIAdem 2010 SP1 which will speed-up loading TDMS files (e.g. containing explicit time channels). We hope to see DIAdem SP1 this year....

[YongqingYe]

Hi yin_the_novice,

Do you find any file size decrement after you defragment the file? And you mentioned the index file is about 75% of the .tdms file, so you write a lot of properties in your file, is that correct?

[Brad_Turpin]

Hi Yin,

At 75% of the data file, it sounds like you're pretty darned fragmented.  I'd say your best bet is to try to defrag the file first.  Can you leave the defragging running overnight?  Have you tried the "DataFileLoad" defrag-while-loading trick in DIAdem?

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[yin_the_novice]

Brad , thanks for your post , i suppose patience is not one of my virtues , I set it running at 1am and it was done defragmenting by the time i checked it in the morning. Works great now.