Max Number of Files to be Unzipped?

[Gregory]

Hi there, for one of my applications the user supplies many CSV files. Right now, the user points to the folder where these are stored. I thought it might be nice to also accept a zip file. However, when I tried unzipping too many files with NI's Unzip.vi or with MGI SZL UnZip to Directory.vi I get an error. It seems to occur with slightly over 2GB worth of files. NI's function will pull all the filenames out, but the files are empty. MGI's function actually throws an error, which is helpful. Through trial and error I found the limit is related to the total size of the files, not just the number of files. The results are summarized in this table:

You can see that the unzipping works fine with 2.07GB of files, but not with 2.21GB. Some example files (where NumFiles = 560 and 600) are attached. Has anyone run into this?

[jordans]

Hi gregoryj,

I think you may be up again a file size limit imposed by the operating system. It is possible that, even though LabVIEW can handle up to 4 GB of virtual memory, the operating system may be limiting LabVIEW to only 2 GB. This KB notes that possible restriction. 

Can LabVIEW Read Data Files That are Greater Than 2 GB in Size?
http://digital.ni.com/public.nsf/allkb/AD5124467245F1D986256B06006F336A

What error are you receiving when you unzip files over 2.07GB?

[Yamaeda · Accepted solution]

I remember reading about some limitations to LV's zip before. I'd recommend running 7zip via commandline instead.

/Y

[Gregory]

Thank you both, I will try the command line route.

Jordans: the MGI function uses .Net calls, so I was getting a .Net exception (1172) , but I forget exactly what it said. If I cannot get the command line to work I'll come back to this.

[rolfk]

This means that the underlaying ZIP routines (at least NIs uses the ZLIB library with MiniZIP extension provided with it) hasn't been updated to support bigger than 32 bit archives. In fact there are several possible limitations here. First the archive itself needs to support more than 32 bit file headers internally, then there is an additional 32 bit limit per file which was addressed with another ZIP extension and last but not least the wrappers using these libraries all have to support it too, by invoking the correct 64 bit library methods.

That the .Net ZIP routines has this problem still too is a bit surprising though. The OpenG ZIP Tools are going to support 64 bit archives including 64 bit files inside the archive soon.

[Gregory]

Rolf: Thank you, that is very informative as always.

Yamaeda: I had success with the 7zip command line tool, thank you!