buffered TDMS file still very large

[niallrobinson]

Hello, I am trying to to stream to a TDMS file. I am writing approximately 10 channels of data at 20 Hz. I was creating a new file evey 5 mins or so and defragging the file once I was finished with it, which was giving me a reasonable size of file. I decided that I would like to make bigger files (over the period of hours) and the the defragging would be too computationally costly to do on the fly. I have been trying to set a buffer for the TDMS files but the resultant files are very large. You can tell the file isn't being writen until its closed (from watching its size in windows explorer) which suggests the file is being buffered. However its about 6 times bigger than an ASCII file of the same data, with and index almost as big again.

Does anyone have any ideas why it would appear to buffer but not actually  reduce the data size. I'm running Windows XP, LabVIEW 10.0.

When I create each new file I run this to set the buffers. (I have also hard wired in a buffersize of 100000 on a one minute file to no avail.

Niall

Thanks

Niall

[YongqingYe]

Hi Niall,

I'm wondering does it make any difference of the file size when setting the property "NI_MinimumBufferSize"?

[niallrobinson]

Hi - I just tried taking data with a buffer size of 100000, 2000 (both of which should be ample) and 10 (which should not). It made no difference to the file size. I suppose this suggests that I'm not setting the buffer size sucessfully which is why I'm getting such large files.

Just to give a bit more detail. I create a new file every X minutes, set its buffest with the routine above, then pass the file reference to a global variable. Once the X minutes are up the file gets closed and a new file is created. Wheneveer I want to log data in the logging VI I just log the data to the file referred to in the global.

Thanks

Niall

[DFGray]

Could you post your storage code, as well?  It would help to see how you are saving the data.

[niallrobinson]

Sorry for going quite everyone - I got it myself. I didn't realise there was an idiot trap when you setup TDMS files and you have to enable buffering by wiring a true to one of the imputs. Why oh why would you disable buffering by defaul. If I got to the trouble of setting the buffer the chances are it should be enabled.

Thanks for your time anyway

Niall

[niallrobinson]

I has this marked as the solution but it turns out it only helped one of my programs. I am still getting very large (fragmented) TDMS and index files for another program. The problem is that if I just defrag the (supposedly buffered) file, it interupts the data logging because it takes so long. I'm 99% sure that it *is* buffereing as it doesn't write till it closes the file, and if I use the read properties function it reads back a set buffer size. Here is the VI that actually writes the data. There bit at the top is for writing an optional acii file so you can ignore that.

Its maybe a bit hard to see whats going on in the next one, but this is where the file is created before being passed to the setbuffer VI which I posted earlier. It also closes that last file

Its really hacking me off now and holding me up from going no to other stuff. It would be great if someone had some ideas.

Thanks

Niall

[niallrobinson]

p.s. the set buffer VI is slightly modified from the one I posted earlier so that it accepts all the data in generic wired inputs. I can post that too, if anyone thinks it might be helpful

Niall

[deppSu]

Hi Niall,

The "disable buffering?" terminal of TDMS Open refers to windows system buffering and it's not related to the TDMS "NI_MinimumBufferSize" property. Thus, I don't think the set of this terminal will help in this case.

[deppSu]

If you still got an index file of large size it means your main TDMS file is fragmented. I've written a simple VI as attached. From the size change of the index file (enable/disable NI_MinimumBufferSize property) you can see this property works.

Can you post a runnable and simplified VI ? Thus, we can look into your VI and see if we can improve your vi to decrease the size of the produced TDMS file.

[niallrobinson]

Ah ok - guess I should apologise to NI about the TDMS buffer rant then! I'm sorry, I can't see an attachment. I should say, defragging the file greatly reduces the file size, emplying it is fragmented. I work on some executable code now and upload it in a bit.

Thanks

Niall