Hello, i need your help for optimization once again.
It's a common problem,
I must write large file results to a sequential file.
The initial data are float SGL type.
To achieve better performance the first idea here is to write into a Tab buffer.
When the buffer is full, it is flushed to the disk, thus enabling better write performance
than the direct I/O at each write.
The second idea is to write data with a size equal or multiple of a cluster size (defines as an external param)
Thus, the buffer is flushed into a multiple chunck of data.
This introduces another buffer level
A third idea is to use different writing componet approach:
- Standard = one by one
- Labview Buffered = Labview component with the previous buffers
- Win32 Buffered = Win32 I/O component with the previous buffers
- Win32 MMF Buffered = Win32 using Memory-Mapped File with the previous buffers
Well Performance mainly depends on HD, but with my computer we can see the results:
(Win32 OS with HD, File size = 120 Mo, Nb Enreg = 2148000)
Standard:
- Read = 21163 ms
- Write = 120202 ms
Advantage/Drawback: Simple to Code but not the max perf
Labview Buffered:
- Read = 9444 ms
- Write = 48618 ms
Advantage/Drawback: Portable, Fast but RAM Buffer requirement
Win32 I/O :
- Read = 9401 ms
- Write = 48618 ms
Advantage/Drawback: Fast but Windows API only
MMF I/O:
- Read = 9623 ms
- Write = 181139 ms
Advantage/Drawback: Slow in Write and Windows Only
From these results, I notice that:
- MMF theorically should achieve the best performance,
in reality, it is not true. Due to page fault and swap ??
- Between Win32 Buffered and Labview Buffered Speed are nearly the same,
with a great advantage of portability for LV.
Thus, I focus on Labview buffered technique and search to optimize it.
Here, the Fill()/Flush() method are working but seem really complicated.
As the problem is recurant in the forum, I distribute the code in order it can serve other developpement.
I use (Standard and Labview Buffered in my programs, no intensive tests on Win 32 I/O, and MMF contains errors).
(See Class FileManager Access)
Can you help me to optimize Fill/Flush method !?
Best regards
Nicolas
