TDMS logging maximum file

[LuckyLuke96]

Hello!

I am fairly new to labview and have just had a project chucked in my lap. The code writes several different numerical values into a TDMS file. The code opens a new TDMS file when the Main VI starts executing, writes to the file in a while loop for the entire duration of the execution of the VI and then closes the file when the program is terminated. I am a little bit worried about a memory overflow when running this program for long periods of time, which is required. Having read other posts in the forum it seems like I would have to write a lot of data (GB range) before this becomes a problem. I do not fully understand what limits the maximum size of the TDMS file. Disk space/file system or RAM? 

I have read: https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z000000PADVSA4&l=sv-SE&OpenDocument=

But I feel like the information is conflicting since it talks about both RAM and the OS file system?

[Ahmed99]

There is function in the RT Module which can compute free disk space so you can estimate when you reach to last memory element so you can hence take decision what to like transfer the tdms file to other big storage and so on

[Hooovahh · Accepted solution]

There's lots that TDMS files can do, and that flexibility can lead to less than ideal files being created.  TDMS files are different from many other file types in that files can be appended to each other and valid files are made.  If you take two MP3 files and put one at the end of the other, you won't end up appending the sound from one file to the other for example.  But TDMS allows for that since every time it flushes data to a file, it also writes the header information associated with the write.  This is a great feature, unless you are constantly flushing to disk, and the header is larger than the data.  Then you end up with huge fragmented files that take forever to open.  My only point with this is design decisions made by the developer might make for a poor experience.  Here's a TDMS presentation I gave at NI Week a few years ago, that covers some of these topics.

But in general I use the RT controller, as a temporary log location.  I make a new log as needed, and then the Windows computer can FTP the files over.  This is to keep the space mostly free on the remote controller where space is at a greater premium.

[LuckyLuke96]

I am not using an RT module only two USB-8520 for CAN communication and on USB-6210 for voltage readings and some digital functions. I have read up a little bit more about the tdms file format and am starting to get a grasp of it. Unfourtunately I have now run it to problems with the logging part of the code. Sometimes the software interleaves one zero value with the actual data causing the output data to have different numbers of samples on different channels. The code is quite problematic using only local variables to write to the logfile, it also uses append array and transpose some arrays before input just so that it can log them. I would like to rewrite all of this but since I got into this project late I don't have time to fix every problem in the code. I can't upload the entire code but here is the section which does data logging.

Could the problem I described be a function of race condition?

[LuckyLuke96 · Accepted solution]

Found the issue, an uninitilized shift register caused a loop to loop to many times when creating the channels!