File manipulation

[JMorgan]

I have a set of data saved to a file that I need to remove a very small amount of information from. The file is binary, but has a ascii header and footer(the header is approximately 13-15 characters, and the footer is 7 characters long). The header is in the format $FILE, XXXXXXXXX; and the footer is READY;
Basically, I need everything between the ; and READY; to be left in the file, but everything before the 1st ; to be taken out.

Any suggestions/Help?

[chilly_charly]

Do you want to wipe out the header and footer from the file on disk, or do you want just to retrieve the info between for processing in memory. If you want to modify the file on disk, do you just want to actually remove the bytes, or just to padd them with some char such as $FF ? What is the size of your file : can it be loaded in memory at once, or is it so large that you have to process it by chunks ?

Lot of questions, corresponding to different possible solutions.

Of course the simplest situation would be to load the whole file in memory as a simple string, then to search for ";" and keep all the chars behind, then search for "READY" and keep all the chars before. Then save the cleaned file.

CC

[JMorgan]

Basically, I just want to strip the header off, and get rid of the last 7 characters of the file. The header and footer are basically just a marker that I use for download control, so they just need to be disposed of. Also, the files could be as large as 600MB or so.....

[JMorgan]

I have figured out how to actually get rid of the header and footer using match pattern, but my biggest problem is the file size. Could you help me figure out how to modify this so it will handle any size file?
Thanks a million!

[johnsold]

If your file size is bigger than the available memory, you have to read it in chunks. Experiment with the sizes which work well for your system.

I woule read a small segment, keeping track of the offset and strip off the header. Write the result to a new file. Read the next segment starting at offset and append it to the new file. Repeat until you reach the EOF (end of file) mark. Extract the READY from that before appending it to the new file. Then delete the old file.

Unless you can be certain that the data patterns in the entire file will never = "READY" your search could terminate prematurely with loss of significant ammounts of data.

[JMorgan]

I guess the main problem i'm having now is that I don't know how to just read in a small segment at a time and keep reading in segments until I reach the end of the file. I can look for the header and READY;(which will only appear once in the file), but since my file can be relatively large, I just don't know for sure how to only read in chunks......

[johnsold]

Use the advanced file functions. They have offset and count inputs where you specify the starting point and number of bytes to read. They output an offset that can be passed via a shift register back to the input on the next call.

You may want to read the fine print and learn how to allocate a big file space on the first write so the OS does not fragment your file as it grows. I think this can be doen, but I havenot worked with files that large and never had to figure it out.

Lynn

[chilly_charly]

Since you already did most of the job, you probably found out, with Johnsold help, how to solve the last problem. Anyway here is a possible solution. After stripping off the header, the file is copied in chunks of reasonable size (10 Mbytes) to a new file. The last chunk is processed to remove the footer before copying.
It is even possible, after some modifications, playing with offsets, to rewrite the file over itself.

You should also know that the footer can be removed without being obliged to rewrite the whole file, just by moving the EOF (End-Of-File) mark. Without header, that would have been from far the best solution...

CC

[DFGray]

If you are working on a Windows platform, your chunk size should be 65,000 bytes for optimal speed. It is a fairly broad maximum, but 10MByte chunks will definitely slow you down a lot and cost you a lot in system RAM when executing.

[JMorgan]

I have one mor situation that I'm hoping you can help me with, seeing as how your last suggestion worked PERFECTLY. I need to do basically the same thing that I did with the stripping off of the header(as far as chunking the file) as I recieve the file from my serial port and write it out to a file. I need to be able to download my data file, which could be 50KB or it could be 600MB. As the file gets larger on my attacehed .vi, I start to lose bits. I have had to change the byte count on the visa read, but still lose data. Could you possibly show me a more effecient way of managing this file transfer for large files?

Again, I am in your debt.

Jdm