Re : TDMS big file size

[qing_shan61]

I have been using .csv files to store my data.  Since I have a long time test to do, I begine to learn using TDMS files to save hard disk space.  But I found it generates two files, one of them is a index file.  Both are big sizes.  Any one is about 8 times greater than .csv file within first 2 hours.  TDMS files are supposed to be small size on disk.  I must have done something wrong?

[EricC.]

Hi

No, you don't do anything wron.

TDMS file have a very hight speed access if they are deframented.

But tdms aren't small.

If you compar the file size and the access time between tdms and binary file you will observe that tdms are bigger and faster as binary file.

regard's

EricC

[qing_shan61]

Hi Eric,

Which is faster, binary or TDMS?

Thanks

[Norbert_B]

Depends.

TDMS is a binary file which uses additional information. So TDMS is slightly larger than binary, but gives you additional information for your data making it more useful on a long term (or do you remember, which column of data refers to a certain transducer if there is no caption on the column?).

TDMS is getting faster if you search for special data sets. The second file you see during creation is most probable the index file which is used by the DataFinder to improve searching in TDMS files.

Regarding your file size: It depends on what version of TDMS you are writing (1.0 vs 2.0). In addition to this, if you write into the file continuously, the file gets fragmented and includes redundant information. You can use the "TDMS Defragment" function to reduce file size. I recommend you NOT to use this function in parallel while writing to the same file.

hope this helps,

Norbert

[qing_shan61]

Hi Norbert,

How can I not to write my data continuesly?  Any experiences?  Any example?

Thank you

[EricC.]

is TDMS more fasted as binary files ?

it depend !!

it depend how you build your binary file.

If you make a simple binary modèle (ex : double) the tdms file is very much faster in writing, and very solwer on reading.

After a deframentation, the tdms is faster in reading.

When you create a complex binary file with size reservation, the binary file is faster.

(increase a file size need a lot of time, so increase it for a big value, write all your date, and at end schrink the file to the good size).

For more a 1 channel it can by comple to developp this binary modèle, but a have tested it and it always more faster as tdms (in read or write and befor or after deframentation).

Other consideration, the TDMS driver lost memory when you don't record datablock with the same size.

ex : you use hight spped daqmx channel, low speed channel and serial data.

ex : 5 channel at 10000 sample/channel/cycle and 10 channel at 20 samples/channel/cycle and 30 channel at 1 sample/channel/cycle

(typiquely SCADA value)

during my test i lost 2Mb/h (i don't remeber the CAR number,sorry).

With same datablock, you have no problem.

best regard's

EricC