Labview 8.20 - "TDM streaming versus Binary Files"

[ashm01]

Hi,

I have multiple loops/parallel tasks with various communication buses in an RT system which need to write data to its own individual log file which is using an Binary format. What kind of benefits will I get in terms of performance by using the TDM Streaming?  The only benefit I foresee is the consolidation of data into an XML type structure and store data into an universal file versus 4 individual ones.

Are TDMS being handled differently in RT for disk IO management? because each write will take upto 8-9 ms.The goal is to reduce disk IO and improve performance.

Any advice?

Ashm01

Message Edited by ashm01 on 10-30-2006 11:46 PM

[BillMe]

My understanding is that TDMS is strictly a binary file while TDM is made of two physical files, one the XML and the other the binary data. My application uses TDM so I can't comment on how good TDMS is, but I think it's optimized to write to disk at high rates (close to the bandwidth of the hard drive interface).

[ashm01]

Hey Bill,

Is your application in RT? I would appreciate if you have some benchmarks on this. I am about to convert the binaries to TDMS and was wondering if anyone had any favorable performance gains.

Thanks,

Ashm01

[Charlie S.]

Hi Ashm01,

The TDMS API was developed to facilitate streaming data to disk at high rates, while maintaing the same descriptive information you would find in a TDM file. You might find the following information interesting from an article on Developer Zone (Introduction to LabVIEW TDM Streaming VIs😞

Unlike the TDM files, which have a strictly required XML-based header file, TDMS files have a binary index file with the extension *.TDMS_Index. The TDMS_Index file provides consolidated information on all the attributes and pointers in the bulk data file, and speeds up access to the data...

I believe you'll find that the TDMS VIs are on par with the regular binary file I/O from a performance standpoint -- plus you get all the benefits that TDM offers. If you are trying to optimize File I/O on LabVIEW Real-Time in general, try writing to a file in chunk sizes that are a multiple of 512 bytes and maintaining a 512-byte-multiple offset with respect to the beginning of the file. Since 512 bytes is the size of a sector on a hard disk (the basic unit that is read or written from a disk), you should see some dramatic improvments.

Hope this helps!

[ashm01]

Thanks Charlie,

I will need to implement to find out the yields. Yes, within the RT we do buffer put mutliple data chunks before we write to disk. However, due to the quantum of data there is some loss of a few ms. Anyways, will post after we see some results.

Ashm01

[CracKatoA]

Hello ashm01. Be carefull with TDMS. Follow this link:

http://forums.ni.com/ni/board/message?board.id=170&message.id=213921

The TDMS is no so good. You would have to continue using Binary Files VIs. The TDMS increase the memory when they save data (See the details in the link). The real problem is we don't know the implementation details. The Binary Files VIs are very good.

[Herbert Engels]

CrackAtoa,

since the problem described in the thread you are referencing is not inherent to the TDMS feature, we do recommend using TDMS.

I highly appreciate your contributions to the TDMS related discussion threads. It's our customers' enthusiasm that drives us to come up with even newer, even better technologies every time we release a new product.

Herbert

[CracKatoA]

Thanks Herbert. Ins't the following inherent to the TDMS features?: "That's why they keep index information in memory while they are writing to disc. Every chunk of data that is written to a channel will add several bytes to that index information." (...for Herbert)

Maybe I misunderstood. Can you teach me? When TDMS finally save the data in the hard disk?:

-every time I call "Write TDMS"

-every time Any buffer (??) is full.

-And for every case: What is Flush TDMS????

Because I understand the binary files VI save the data every time I call "Write binary file" (If we look the windows explorer, the size of the file is growing up while the application is runing)

I highly appreciate your contributions to the TDMS related discussion threads.

[Herbert Engels]

CracKatoA,

the TDMS functions immediately save their data, just like the binary functions. However, "saving" means that your data goes into some buffer owned by the operating system, and the operating system will decide when to actually write data to disc. The "flush" function forces the OS to write to disc immediately, so even in case of a power outage or otherwise complete system failure, your data is on disc. This is a rarely used feature, but some of our customers rely on that capability. BTW, we also have a flush function for binary files. It's on the "File I/O -> Advanced" palette.

You're right that keeping indexing information is inherent to TDMS. The amount of memory consumed in Risotto's app is higher than expected though, that's what I was referring to.

As for the index information itself:

• The good news is that there is a workaround if you run into this problem, which is writing larger data chunks.
• The bad news is that we do not currently have a way of stopping LabVIEW from keeping that information. Maybe we can add a "write-only" flag or something of that sort in a later LabVIEW version, so streaming does not have any memory problems at all. Obviously, I cannot commit to a time of version or to the exact design of a solution like that.
  

Hope that helps,
Herbert

[CracKatoA]

Thanks Herbert!!!

I have not doubt now!!