LabVIEW
Open Large TDMS File Results In File Deletion
I have a large TDMS file that gets logged to by code running on an embedded Linux RT controller, cDAQ-9136. It logs this data to a USB drive formatted in NTFS. The resulting TDMS file size is roughly 4.2GB and the index file is 14MB. After the test was done we pulled the USB drive out, plugged it into our Windows computer and went to do some data processing.
Using a normal blank VI I can call the TDMS Open function followed by the TDMS close. The Open generates an error 6 Generic File I/O error immediately. After generating this error the TDMS file is gone! Using some undelete software we were able to recover it and from that point on we decided it would be best to copy this 4.2GB of data to our local hard drive then run post processing code there. The result is the same with this recovered log, and other new logs. Calling the TDMS open generates an error and then the file is deleted. In one case I noticed that a .tdms.log file was generated with the following text in it:
TDMS: ERROR: TDS Exception in Initialize: Tds Error: TdsErrFileCouldNotBeOpened(6):
I assume this was generated by the TDMS API.
I opened the TDMS file in a hex editor, and then deleted roughly the second half of the file, stopping at a chunk (a section starting with 0x54 44 53 6D). This first half of the log appears to open just fine and regenerates the index file and can be processed. This leads me to believe that there is some TDMS limitation that I am hitting which is causing an error and file deletion. From what I've read the file size shouldn't be the issue because it is on an NTFS drive both on the USB or when copied locally.
Looking at the number of samples in a channel I see that the first half of the file has 2,242,727,592 samples, meaning the full file has roughly 4,485,455,184 samples. The offset and length inputs to a TDMS read are I64 so I assumed that meant I could address the full range of I64 positive values (9,223,372,036,854,775,807). All my searching on TDMS limitations say I'll be limited to the harddrive size before anything else but clearly this isn't the case.
So first, what limitation could I be hitting, and is there a way to avoid it? And secondly why is my file being deleted on a File I/O and can if this is a bug in the TDMS API can this be fixed?
LabVIEW 2018 32-bit is used on Windows 7 x64, with all other toolkits and sofware coming from the 2018 developer suite DVDs. I don't mind sharing this 4.2GB file but obviously it will take a while to upload, and download it. Does NI have any suggestion on how to share the file?
Also just a little bit more information if you've made it this far. The reason for so many samples in a channel, is because this is a CAN bus log, and NI has made a standard for logging raw CAN frames by flattening a single CAN frame into an array of 24 bytes and writing that to a single TDMS channel. With a standard CAN bus at 500Kbaud, and logging for multiple days, the number of samples in a channel get quite large. I'm open to creating my own TDMS CAN log standard made up of multiple channels, but then it won't mean being able to use the DIAdem Bus converter, not that I can do that with these large files anyway.
I guess LabVIEW 32-bit has a file size limit of 4GB? Still yet, it shouldn't delete your file.
Just went through the trouble of creating a 64 bit virtual machine, then installed LabVIEW 2018 64-bit and the result is the same. The TDMS Open throws a generic error and then the file is deleted. Pretty annoying.
I also went through and created a 4.6GB random TDMS file with a similar amount of data in a single channel. It has 4,800,000,000 samples on a single channel. In LabVIEW 2018 32 bit I can open, query and close that file with no problem. This likely seems like a problem with this TDMS file which as far as I can tell is a valid file.
@Hooovahh wrote:
Just went through the trouble of creating a 64 bit virtual machine, then installed LabVIEW 2018 64-bit and the result is the same. The TDMS Open throws a generic error and then the file is deleted. Pretty annoying.
I also went through and created a 4.6GB random TDMS file with a similar amount of data in a single channel. It has 4,800,000,000 samples on a single channel. In LabVIEW 2018 32 bit I can open, query and close that file with no problem. This likely seems like a problem with this TDMS file which as far as I can tell is a valid file.
Thanks for going through all that trouble to prove me WRONG - LOL! 😄 Even if there was some kind of fatal flaw with the formatting of your file, LV shouldn't eat your file. LabVIEW eating a file because it doesn't like it sounds like a bug to me. 😉
For me the motivation wasn't just to prove you wrong, but for my own sanity as well. I mean why should a 32-bit application not be able to use an I64 as an address? Anyway I have opened a service request with NI and they are investigating it. If I get a CAR or if NI tells me I'm an idiot and did something wrong I'll update this thread.
