TDMS Open: What does it DO?

[10Things_Rob]

So I'm trying to debug a crash on an sbRIO.

The RIO takes data continually and periodically writes various data into TDMS files.

For space reasons, I am using these files WITHOUT index files.

For other reasons, I am not leaving the files open when I'm not writing to them - instead I open, write, and close when I write.

I have observed that in the Distributed System Manager, when I open/write/close TDMS files, my CPU spikes up from about 70% to 99%.

Not only that, but the longer I let my system run without resetting the files, the longer (in time) the CPU spike is. So let's say when I first start, I get a spike for about a second. Two hours later, the CPU is at 99% for six or seven seconds.

My hypothesis is that when I do a TDMS open, even though (or because of the fact) that I am not using an index file, the TDMS open is creating an index in memory before allowing me to write to the file.

Is there a TDMS expert out there that confirm/refute this hypothesis?

It feels to me that if I am writing data to a file, I'd like to be able to just WRITE to the file. I know when I open it that I don't want to read it. Is there a way to open the file as "write-only"?

I haven't dug too deep into the structure of the TDMS file lately, but it seems to me that if you have to modify data in the existing File Data in order to append data to the file, then there's a good reason for a TDMS 3.0. If you DON'T then there's a need for a "Write-only" option on the TDMS File Open function.

[BertMcMahan]

Do you have "Create index file" set to F when you Open it? If that's T then I'd guess you're recreating your index on every Open.

[10Things_Rob]

Yes, that input is set to FALSE.

[10Things_Rob]

... and I've proven my hypothesis.

What I did: Open-Write-Close, timing each operation. This was on a very old sbRIO.

Each time, I wrote 4808 bytes of data into the file.

I repeated this 200 times on the same file, starting with an empty file.

What I found was that the time required to Write (~7ms) and Close (~26ms) was pretty consistent.

...and then there was Open. It climbs linearly with the file size.

If you use the index file, the 200th Open operation takes ~370ms!!!!!

...and if you DON'T use the index file, you're talking  ~550ms!!!!

Wow, so I am open to any workarounds, but I am shocked at how bad this is.

Any insights NI?

[cordm]

You can try using the TDMS Avanced Functions, the help explicitly states that TDMS Advanced Open does not create a tdms_index file.

Can you avoid reopening the file? If you want to ensure data is written to disk, you can use TDMS Flush and don't need to close it.

[10Things_Rob]

The way I am using TDMS Open (or TDMS Advanced, doesn't seem to make a difference), it does not create the index file. At least on disk. It seems to be creating it in memory, which is likely what is taking so long.

If I recall, the reason I am not reopening the file is that in the past, if I left my file open and lost power to the RT, then the entire data file would be corrupted. I have not tested if that is still true.