TDMS flexibility / performance

[emmazheng]

Thanks for your validation. It is true that TDMS Advanced Synchronous Write performance can be affected by increasing number of TDMS segments in file (TDMS Advanced Synchronous Write is irrelative with a .tdms file of growing size and fixed number of TDMS segments). The improvements are under research. Currently please use TDMS Set Channel Information before multiple continuous synchronous writings. Jie Zheng National Instruments R&D

[Intaris]

---

@emmazheng wrote:
Thanks for your validation. It is true that TDMS Advanced Synchronous Write performance can be affected by increasing number of TDMS segments in file (TDMS Advanced Synchronous Write is irrelative with a .tdms file of growing size and fixed number of TDMS segments).

---

Any specific reasons why this is so.  To my naive viewpoint, this should not be the case so it would interest me if it's just a missing optimisation or if it's a fundamental limitation of the file format.

---

emmazheng wrote:The improvements are under research.

---

Given the trouble I had getting my message across that this problem even exists, would it be possible to be included in the testing of any improvements made?  I'm already NDA'd up to my eyeballs.

---

@emmazheng wrote:
Currently please use TDMS Set Channel Information before multiple continuous synchronous writings.

---

I already do this (or at least this is what my software SHOULD be doing).  Is this meant as a fix to the problem somehow?  I don't see the connection at all, sorry.

Shane

[emmazheng]

Hi Intaris,

As to your main VI, if you keep set channel information function and synchronous write function in a loop, no matter outside loop or inside loop, the writing performance would not be improved. Currently I suggest to separate TDMS Set Channel Information from any loop directly or nestedly includes TDMS Advanced Synchronous Write. 

In order to make a clear demonstration, I wrote two VIs to compare the writing performance. Both of the VIs create a .tdms file including four channels and every channel includes 1,792,000 samples. The performance graphs are as below.

Original performance:

Improved performance:

The 2012 version VIs are attached.

Jie Zheng

National Instruments R&D

[Intaris]

OK, I finally got an update on this issue.  Thanks to NI Support (Julien) for passing this on.

1. ---

   The advanced write function not only writes raw data but also the lead in and meta data. The advanced write function needs to find the previous segment offset in file then modify the next segment offset in meta data of current segment. Currently, the advanced write function searches TDMS segment from the beginning of file to locate the previous segment. That’s why advanced write gets slower with more segments.
2. The key issue affecting writing performance is not that the advance writing is relative with segment number, but that too much segment meta data need to be flushed to disk. For a segment, time for flushing meta data may be more than 1000 multiple of time for flushing raw data. TDMS advanced functions were designed based on real use cases identical with what the examples show. In my opinion, not pre-defining all the channels is not common use case for TDMS advanced functions.

   ---

I was under the impression that Advanced Write did NOT write meta data, this is why I was trying to use it in this way.  I think the information given here is a bit misleading.

Compared to the standard TDMS VIs and functions, the Advanced TDMS VIs and functions have the following advantages:

• Writing meta data and raw data separately—The standard TDMS VIs and functions write meta data and raw data at the same time. The Advanced TDMS VIs and functions enable you to write meta data and raw data separately. You can use the TDMS Set Channel Information function to write the meta data into a .tdms file, and then use the TDMS Advanced Asynchronous Write function to write the raw data into the .tdms file.

The advanced write function should (afaik) always write to the END of the file (unless I have been fiddling around with the write position).  Having previously declared which group / channels I am writing to, I assumed (wrongly apparently) that the driver would cache any offset information it might require for the operation and certainly would not have thought that the function would make a linear search through the file in order to add a byte at the end.  That seems a bit..... inefficient.

So either way, the tdms implementation is broken for my application.  In order to stream information to disk, it would appear that much more work is being performed than would otherwise actually be neccessary.  But that's actually my problem simply because I don't have a "common use case for TDMS advanced functions".   Well guess what.  Since TDMS doesn't WORK for my application, it'll probably be staying that way.

Back to the drawing board.

[Intaris]

A quick thought which just occurred to me.

What does the "Set next Write Position" actually do?  Does this allow me to circumvent the entire "Search the file linearly" aspect?  Would this work if I had previously written data in one go and then choose to overwrite it bit by bit?  Would this help get around the problem with the number of segments?  It's kind of defeating the purpose of using TDMS at this stage, but itmight rescue enough to make a new attempt worthwhile.