On multiple starts my Application sometimes slows down dramatically on File I/O

[Hacky]

I wrote a small application that converts some Log output to several type of application specific file e.g. .csv for importing it to EXCEL or DIADEM.

The task is very simple, just reading a file sequentially line by line and scan the contents into several variables. If the scan succeeds

write the contents into several output files.

Normally the mean file access time is 0.16 ms per Line.

In 10 to 50 percent of the program runs  the file access time increases to a mean of 9.5ms and which means each file access` time is a minimum of 9 ms. See attachement. The File is located on a DFS Share and is > 20MByte in size.

Moving this File to a local Filesystem e.g. C:\F\..... changes the file access time to a constant mean of 0.14ms per Line.

- It is not applicable to have the file on a local drive so i need a solution for this effect.

- Ofcourse there are workarounds using a paged file read, but for such types of small applications the programming overhead is considerable.  

Does any one have an idea regarding the reason for this behaviour?

[TCPlomp]

Do you open the file for every read? or just once and just read a new line every time?

The second method should be considerably faster

Ton

[Hacky]

Hi Ton,

thanks for the Idea but ofcourse i don't open the file every access. the file remains open after an initial access and is read sequentially.
There is no file positioning before the read because the file position is maintained after the read.

There is also no general timing issue, as i wrote:

- from local system system the access time is 0.14ms per line which results in an amount of 1 minute per 10 MByte of data.

- from DFS file system there is a spread between 0.16ms per line which is a good performance 
  up to 9 ms per line and more which leads in an amount of 30 Minutes and more for 10 MByte.

Thanks for the answer, Hacky

[Hacky]

Hello again,

i should have enclosed my small (and dirty) application. Its just an ad hoc converter for file contents.

Also some test data.

regards, Hacky

[TCPlomp]

Well the problem seems to be the transfer. What you could do in your code is read 1000 lines at once and in a for loop process the data. This will lower the numver of times you acces the drive

Ton

[Hacky]

Thank you Ton,

thats what i know as workaround for thr time beeing.

Best regards, Hacky