How you do stop a large file becoming un managable

[alecjcook]

This simple program will create a file filled with sequencial 8 digit hex codes seperated by commas, ie.. (AA0AAAAA, AA0AAAAB, AA0AAAAC etc...)
The problem I have is when running this program, the more values are added to the file, the slower the program runs... I hope to itterate this approximatley 100'000 times.

The first 500 values are placed in the file quickly, but each subsiquent value takes longer and longer. I assume this is because the file is taking longer and longer to access... Does anone know what route would be the best was around this crippling perfomance issue?

Many thanks, Alec

[smercurio_fc]

Your problems stems from the fact that you're using the all-in-one Write VI. This VI opens the file, writes out data, and closes the file. This is an expensive operation and just gets costier and costlier in your case. The right way to do it in your case is to open the file once before your loop starts, and then write the data to file within the loop, then close the file when the loop ends (either when you end your iterations, or you get an error). This means you have to use the "regular" Open File, Write File, and Close File. Check the examples that ship with LabVIEW. You'll find a number of File I/O examples.

[GerdW]

Hi Alec,

in addition to smercurios tips I would:
- collect data to a reasonable string size of (let's say) 4kb instead of writing 9 bytes in each loop iteration
- use "format into string" with format string of "%08x," and one "write to file" instead of your formatting function and two writes to the file
- get rid of that sequence structure, it's not needed here!

If the computer is a modern one you can also create the string in RAM and just write once to disc (100.000 times 9 bytes gives a file of ~900kb size - quite easy to handle nowadays).



Message Edited by GerdW on 01-10-2008 04:16 PM

[GerdW]

Hi Alec,

here comes an example (in LV7.1)...

[DFGray]

Note that the optimum block size to write or read a disk on a Windows system is about 65,000 bytes.  Double buffering is your friend.

[DFGray]

Note that the optimum block size to write or read a disk on a Windows system is about 65,000 bytes.  Double buffering is your friend.

[GerdW]

Hi DFGray,

I used this technique on an older Win95 system with limited RAM (and processor speed), that's why I chose 8kb...

[Ben]

Pre-allocating space for the file also helps with file write performance.

Ben