TDMS flexibility / performance

[Intaris]

Questions upon questions regarding TDMS files.

I am trying to write a set of data with (for examples sake) 10 groups with 20 channels each with 8192 data points each (1pprox 13MB per dataset).

Even when using the Advanced functions and enabling or disabling buffering I am seeing significant slowdown with increasing file size.  As soon as I start adding more and more sets of data, the file writes slow down considerably. 3.7seconds to write.

If I repeat the above dataset a few times the write time rapidly rises to over 30 seconds after approximately 5 writes (appending to existing data).  That's 10 groups with 20 channels each and 40960 data points each.  As SGL data type that's only just over 65MB of data.  Not what I would call a particularly large file.

How does TDMS handle this internally, it would seem that each new set of channels added to the file metadata slows down all subsequent writes a little.

[James_McN]

It could depend on how you are writing. If subsequent writes contain different channels then it rewrites the meta data and has hold a model of this in memory.

That said this shows a memory increase but don't recall seeing a major decrease in write performance.

This article helps understand how it works http://www.ni.com/white-paper/5696/en/

Perhaps try playing with buffering on or off? The rates you are seeing funny seen like the hard disk would be a limit and set up right I have run tdms at 100's mb/s

[James_McN]

Sorry, just reread your post and saw you have tried buffering!

[James_McN]

Sorry, terrible answer, actually in front of LabVIEW now and a quick benchmarking app shows similar things but I see a step change, about 100-200ms per write and then a lot of jutter running closer to 1 second per write. Is this consistent with what you are seeing?

The first thing I would suggest is try to write to one group. This prevents it having to rewrite metadata every time. If you look at the advanced API examples that is basically what they are doing, giving you control over writing the meta data and then the real data.

Quick switch to this still seems to show these variations as well in my benchmarks.

[YongqingYe]

TDMS streaming speed should be irrelevant to the file size. What is the file size when you observing the speed slow down, hundreds of meta bytes of even tera bytes?

And it would be very helpful if there's any VI or even screenshot of the VI to be post here to reproduce the problem. 

[Hooovahh]

I've been somewhat following your other TDMS questions, but haven't found many places to help, due to code not being available.  I know that might be difficult but honestly how data is written in TDMS matters a lot.

Attached is a quick VI I wrote.  Every time you press a button on the front panel it will write to 10 groups, each with 20 channels, each with 8192 samples of the numeric single.  On my machine after performing 20 writes I ended up with a 128MB file, and a 150KB index.  And the maximum time to write those 10 groups, 20 channels, and 8192 samples was 24ms.

TDMS has the performance that you need, but understanding the code you have written, will show why the slow down happens, when it doesn't in other cases like this one.

[Intaris]

I've been away so sorry for not answering sooner.

I'm attaching some code (saved for LV 2012) which displays the problem I am seeing.  The VI includes data in a form we receive it which we want to stream to disk.  It's highly fragmented data and we currently require a lot of individual calls to the TDMS write function.  I had a version where we buffer our data before writing, but performance was not better.

I am using the advanced TDMS functions because I want later to be able to re-access previously written data piece by piece (We may not be able to hold an entire dataset in memory).  I also have not disabled buffering in this example.  I had a version where I was actually buffering the data myself but this was significantly worse, no idea why.

On my machine, I see an entire iteration of my data (10 groups, each with 20 channels of 16384 datapoints) taking a baseline of approximately 3 seconds.  Each iteration of appending the same structure to the same file shows an increase in write time of approximately 100ms.  I have also tries switching off the meta data writing and only writing the measurement data (accumulating all of the data int eh last-written channel) but the increase in write time seems to scale with file size (try switching off meta writing for a while and then re-activate it).

I'm at a loss at the moment.

[Hooovahh]

I haven't full dove into your code but here are a few comments:

---

@Intaris wrote:

I am using the advanced TDMS functions because I want later to be able to re-access previously written data piece by piece

---

Why is the advanced functions required for this?  You can use a TDMS read on the same reference as the written one.  I've seen it done in parallel without issue too.

One reason why I think you have increasing time, is you are closing and re-opening the TDMS reference each iteration.  Why not open once, then close once?  Even so I did a quick test and the time still increases with a open once and close once so I guess this isn't it anyway.

[Intaris]

Opening and closing the file is not the problem.  I have even tried turning off the meta data (Channel information) and just dumping all of the data into the last defined channels.  The write speed stays constant (but the data is all int he wrong place of course).  When I re-activate the meta data, the write times increase not from where they were the last time I wrote meta data, but as if the file size itself were the limiting factor.  I'll try to get an image later.

I've tried all configurations but I just can't get

a) the speed back to where my initial testing let me expect

b) rid of the increase in write time

Shane.

[YongqingYe]

From my personal perspective, the VI attached is still too complicated. Is it possible to get a much simpler VI just to reproduce the problem of TDMS performance you described, without any queue, case structure, and so on?