Retrieving large amount of saved data

[MogaRaghu]

In a product test rig application, I am saving a large number of test results and also parameter limit values for declaring pass / fail verdict. The saved data resides in an independent file on the hard disk with the extension .vwe.

When the user wants to view the saved data along with the test graph, I am retrieving the information from the hard disk on to local variables. That portion of the code alone is presented in the enclosed VI. Just to save on the server space ( and your time ) I have removed all other blocks from the scheme. So its a broken VI!

I just want to know if there is any better method of doing the same thing. As the number of data being retrieved is large, making the scheme layout is a big challenge. So it is not very neat - but if this is the only way, then I will have to make it neat.

Kindly throw some light on other, elegant methods if any.

Thanks

Raghunathan

[DavidT]

Well, first thing, your block diagram is pwd protected! Can you please remove the protection so that we may take a look?

[Joe_Guo]

Here are some tips I used frequently when retrieving large data sets:

0. NEVER read large data in one operation
a. decide the start point, data size and (downsample points if needed) to read
b. if the data size is too large, read the data in small chunks. How large is a chuck is personal preference.
c. put a 1 or 2 msec delay every several reads if the data is really large to reliefe the stress on CPU.
d. if the data is for displaying, the data is normally downsampled (either read one line per x # of lines, or do an everage).
e. get the data channel needed (normally only few channels are needed at any moment).
f. if the data is for conversion, conver the data after each read.

I used these steps on files over 1GB (2G is the filesize limit for normal labview application, and requires more considerations. I normally limit the filesize to below 2G per file), the speed and memory footprint are pretty resonable even for a lower end PC.

-Joe

[MogaRaghu]

---

@DavidT wrote:
Well, first thing, your block diagram is pwd protected! Can you please remove the protection so that we may take a look?

---

Oh Oh I am so sorry!

Enclosing again with password protection removed.

Kindly revert on some tips to do what I have done more elegantly.

Regards

Raghunathan