LabVIEW
Append to a compressed file
@mister Rose wrote:
I have some test results that I want to save into a test result file. Since I will be accumulating many test results, I want to keep overhead down, so I would like to simply append the result to the end of the file. Here is what I have done...
1) Create an empty array, flatten to XML and write to a file.
2) When I want to add a record, I take the record control, flatten to XML, and then overwrite the </Array> at the end of the file with the XML record. I then append another </Array> onto the end of that to properly close it out.
3) I can then read this XML file back into an array of test record structures at a later time.
I chose to do it this way, because when I am generating the test records, I do not want the overhead of "file read->XML Parse entire file->append code->XML export appended structure->File write". I figured that by simply appending the XML record to the end of the file, I could cut down on a lot of this IO overhead.
The problem is that XML is a bit "chatty". I'd like to be able to read, write and append to compressed files (since XML compresses very well). The lvzlib routines only have a read and a write, but no append. My understanding of the compression schemes is that it makes a table of the common byte patterns and replaces them with shorter tokens. If I compress each record, I will have a table of tokens for each record that will basically be the same as all the other records. I can get better compression if I use one table for all records.
Has anybody implemented this, or do I need to dig into the guts of zlib?
Alternatively, can the normal File I/O routines be configured to work on a file in a compressed archive (like a zip file) and let the OS handle all the compression?
Thanks,
I'm not sure what you mean by appending. If you mean appending new data to an existing file in a ZIP archive this can't be done in any way. The ZIP file is not structured in a way that would allow appending data to an internal file stream inside the archive. I'm not even sure there is any compression scheme that would allow that. The only way for all standard compression schemes for this is to extract the file append the new data and then replace the existing file in the archive with the new one.
If you mean appending a new file to an existing archive then the newest released version 2.3 of lvzlib can do it. The ZLIB Open Zip Archive.vi function has an append mode input. Previously this was a boolean that indicated if a new file was to be created or if the archive should be tacked to the end of an existing file. In the new version this is an enum with an additional entry (adding to existing archive).
Rolf Kalbermatter
Message Edited by rolfk on 10-04-2007 09:21 AM
@mister Rose wrote:
I was also considering compressing each record. I should be able to do this with the Zlib code.I could also save each test record as a file and Add it to a Zip file using the Add to Zip File. I have to see if I can go straight into the Zip file, or if I need to save it to a file first, then Add to Zip, then delete the file.Many Options.I'll try to post what I find.
I'm currently (on and off) working on a new release of the lvzlib library that will allow both in memory archive streams as well as adding memory buffers to an archive (file based or in memory). The code compiles and runs fine on Windows and also on Mac OSX but has still some crash in the shared lib under Linux. When I get to squash that crash I will try to release a new version of the library.
It still won't make the file streams inside an existing archive streamable in itself. This would require some serious memory shuffling in the lvzlib library and I'm not sure I can see the benefit of this in comparison to doing an [extract file, append data, replace file] externally, using the LabVIEW VIs.
If you want to see the VI library you can get it from the CVS repository inside the OpenG Toolkit on sourceforge.
Rolf Kalbermatter
Message Edited by rolfk on 10-04-2007 09:33 AM
