buffered TDMS file still very large

[niallrobinson]

Hi - I've attached a bare bones version of the program which has the same problem. The "logger" vi is the main one. I've also attached some sample data. Thanks for the help

Niall

[blawson]

When I open the tdms file using the tdms->excel importer, I see that only two channels in each group have NI_MinimumBufferSize set (to 3000).  The rest are defaulting to instant writes, creating a header line at every write, blowing up your file size.

your "set buffer.vi" wasn't in the attached zip.

[niallrobinson]

It turns out that the channel array needed transposing (a problem with the bare bones conversion) which I have done in this version. I have also included the setbuffer file now. I didn't realise you could see the buffer size in the excel file

Thanks again

Niall 

[blawson]

which begs the question: do your production tdms files show the buffersize property for all channels?

[deppSu]

Sorry for forgetting post my example VI previously.

I have tried your VI. The main cause it has is that it uses "interleaved" as the data-lay out input for TDMS Write. If the data to write is interleaved TDMS has to flush its buffer for each write call thus the minimum buffer size property can't take effect. The reason you see that the data seem to be "buffered" by TDMS previously is because TDMS has a default  2MB disk cache ("NI_DiskCacheSize") for "disable buffering" write. If you set the "disable buffering" to false, you will see the file size is continueously increasing by refreshing the explorer.

[niallrobinson]

OK - so that explains why the minimumBufferSize didn't work then. Its seems odd to me - would the main time when you want to use a buffer be when you are writing interleaved data i.e. streaming real time to a file? If you can only use the buffer on non-interleaved data then you would hardly get any fragmentation anyway because you would only do as many writes as the number of channels. Unless I've misuderstood.

Do you have any recomendations for a better way to do this then? Is my only option to assign an array and then fill it up and write that to a tdms? I though the whole point of the buffering was to avoid that situation.

Thanks for getting to the bottom of that

Niall

[deppSu]

From LabVIEW 2009, TDMS have a feature called "one header only", that means if you keep writing data of same data-layout ( same channels, same number of samples and same data type), your TDMS file will save "header"s and decrease file size. like the following:

If you can re-design your logging part ( try to separate your TDMS Writes from one loop and make a loop for each write) and make it fit "one header only" requirements your file size will be greatly decreased.

[niallrobinson]

Thanks very much - I'll give this a go and let you know if it works.

[blawson]

---

@deppSu wrote:

The main cause it has is that it uses "interleaved" as the data-lay out input for TDMS Write. If the data to write is interleaved TDMS has to flush its buffer for each write call thus the minimum buffer size property can't take effect.

From LabVIEW 2009, TDMS have a feature called "one header only", 

---

Where are these things documented?  I am continually frustrated with how incomplete the TDMS reference materials are.

One header only: does this mean identical calls to TDMS write?  Not very useful if you need timestamps and data.

[niallrobinson]

I completely agree with blawson about the help files. This is my first labVEIW program and I am amazed by how lacking the help is in general (not on the forum obv - its excellent here ;). How a visual language like this doesn't have pitures in the help is beyond me (I know there are examples).