Max binary or TDMS file size to read (64-bit)

[zz_candy_fan]

I am trying to read either binary or TDMS files that are larger than 4 GB, but labview failed to do this, and showed Error - 2501 (possible reason: invalid TDMS file reference).

The Labviews we are using are 2013 and 2014, 64-bit. Both of them cannot read files larger than 4 GB. 

The computer we are using: Windows 7 professional, Intel Core i7-4770, 3.5GHz, RAM: 32 GB, 64-bit. 

To genarate and read the binary and TDMS files, I used the examples provided by Labview: Write and Read binary file.vi, and TDMS Advanced Synchronous Write and Read Speed Test.vi. 

Could someone help me to figure out the problem?

[Taki1999]

Try reading the file in chunks. (i.e. wire in a count of 1000 instead of trying to read the whole file)

Also are you sure your files are actually TDMS or are they a different type of binary? It's not clear from your post.

In general, it's not a very good idea to generate 4GB files. Usually there's a better way of chunking or segmenting your data.

[zz_candy_fan]

Thanks for replying. 

We need to acquisite data more than 4 GB. This is the bottle neck we are trying to remove. 

I tried both binary files and TDMS files (Example TDMS Synchronous Speed Test Data.tdms). I am a beginner and I do not understand the chunks very much, but is it similar to the blocks in the example .vi?

[Taki1999]

My LabVIEW is only 32-bit so I get memory full errors whenever I try and work with a TDMS greater than 4GB so I may not be able to help too much.

Does something like this work on the read?

[zz_candy_fan]

Thanks for replying. I finally found out the reason. The defalt datatype of the channel length from TDMS Get Properties Funcion is I32. It has a maximum value of 2147483648. After I change it to I64, it is able to read files that are larger than 4 GB. 

[Taki1999]

Excellent! Glad you found that part out.

I still think you may want to think a bit more about your data organization to avoid 4GB files though.

[Sam_Sharp]

Agree with Taki - I would strongly recommend that you split the files up in your data acquisition system - I would start a new TDMS file periodically (after X points, after the file reaches X MB, after X minutes)  to avoid the large file sizes. Having smaller files will make them much more manageable - for opening / processing / transferring etc. It also means that if something happened like the file becoming corrupted (e.g. power off, bit rot, cosmic rays), you'd lose less data.

[zz_candy_fan]

Thanks for your suggestion. Is labview able to do that automaticlly (after some centen time or when the file reach a certain size)? If it is, is there any similar example .vi that you could recommed for me, please?

[crossrulz]

---

@zz_candy_fan wrote:

Thanks for your suggestion. Is labview able to do that automaticlly (after some centen time or when the file reach a certain size)? If it is, is there any similar example .vi that you could recommed for me, please?

---

Where is the data coming from?  If you are just logging data from DAQmx, you could just use the DAQmx Configure Logging and then set a property for how many samples go into a file.  DAQmx takes care of everything for you from there.

[Hooovahh]

Yeah if it is DAQmx writes you can configure it to make new files.  If it is being done with TDMS primatives, then you can do things like check file size or a timer periodically and make a new file as needed.  LabVIEW is a programming language, you can make it do just about anything.