Freeing memory when loading large binary files into memory

[dwj314]

I am loading a 200 MB binary file into an array of clusters (each cluster contains 2 U32s, 1 single, 1-D array of U16s, and a 1-D array of singles) to allow users to go back through data that has been collected.  However, I see massive amounts of memory being used and I don't exactly know why it isn't being freed.

I ran some tests with just the data loader to try and reduce the memory usage of my application.  In the first test I do not output the array to an indicator; however, I need the indicator because my loader is a subVI and I need to pass the loaded data back to the main application.  I tried to use a reference to the array but that ended up using more memory overall (which I also don't understand).

I used the PF Usage in Windows Task Manager to collect these numbers. The LabVIEW profiler shows my VI using 210392 KB in the Max Bytes field for the second run below.  Subsequent runs show double that; I suspect this due to the pre-allocated array being copied into the already filled indicator.

The same data being loaded via the reference shows 674020 KB in the Max Bytes field!  I definitely wasn't expecting that.


---------------------

without output array to indicator

start: 907 MB
end: 1.09 GB = 1116 MB

diff = 209 MB   (as expected since there is no indicator)

----------------------
With output array to indicator

start: 890 MB
end: 1.28 GB = 1310 MB

diff = 420 MB

---------------------

Pass a reference to the loader and have indicator in the main app

start: 1.04 GB
end: 1.52 GB

diff = 491 MB

----------------------

I added the Request Deallocation VI per some suggestions I've read online but that does not do anything in all these cases.

I figured pre-allocating the array in the loader and then replacing each element in the array would help with the load time:  all the required memory is set aside before I start throwing data into memory.  However, I end up with 2x the amount of memory being used as I expected and even more if I use a reference to the original array and remove indicator in the loader.  How come?

-dennis.

[Jason_P]

Is there any reason you are loading the entire file at once?  With the structure you describe, it seems unlikely a user would need to see all 200MB of data at the same time.  With a binary file, it should be easy to write a subVI that could pull just the subsets of data the user needed at that time. 

[Andrew_E]

Hi Dennis,

First, I am not sure why you are dumping the binary file to an array of clusters? Are you searching by clusters? Why are you not searching the file itself for this information.

For the massive memory usage, try initializing the array before you add any of the clusters to it. Also are you passing this array through any tunnels? This could create a copy of your data. Are you splitting the array data path any where?

The deallocate memory vi only works within subvi's after they are done executing.

Regards,  

[Ben]

---

dwj314 wrote:

I am loading a 200 MB binary file into an array of clusters ...


---------------------

without output array to indicator

start: 907 MB
end: 1.09 GB = 1116 MB

diff = 209 MB   (as expected since there is no indicator)

----------------------
With output array to indicator

start: 890 MB
end: 1.28 GB = 1310 MB

diff = 420 MB

---------------------

Pass a reference to the loader and have indicator in the main app

start: 1.04 GB
end: 1.52 GB

diff = 491 MB

----------------------

I added the Request Deallocation VI per some suggestions I've read online but that does not do anything in all these cases.

I figured pre-allocating the array in the loader and then replacing each element in the array would help with the load time:  all the required memory is set aside before I start throwing data into memory.  However, I end up with 2x the amount of memory being used as I expected and even more if I use a reference to the original array and remove indicator in the loader.  How come?

-dennis.

---

The extra data with the indicator form would show-up if you had the FP of the sub-VI open. LV has to keep a copy for the indicator so you can look at it while watching it in execution hightlight on the next iteration etc.

Please review the "Clear as Mud" thread for a mind-boggling explation of when and how buffers ca be re-used in sub-VIs.

Ben

[dwj314]

The data ends up being played back like video -- the saved data is fed back into the displays so the user can review the data; there they have control over the playback speed, direction (forward/backward), and can use a scrub bar to jump around within the data.

There's two reasons for loading the entire file into memory: the saved files are often archived to the network and then reviewed later (days or weeks later) and I have a tagging system in place during data collection that lets the user jump back into saved positions during playback. 

[dwj314]

Hey Ben-

I was getting the same results with the sub-VI closd; I ran across that "Clear as Mud" thread while researching before posting my question which was why I was still baffled.

-dennis.

[dwj314]

I put some more details of what I am doing with this file in the reply to Jason P. above: basically, the data in the file is a time-based data stream.  I load it into an array so I can index into it quickly during review -- either by using a tag that was created when it was recorded or with a scrub bar (like in video editing).

For the examples I posted, I don't have any cases or tunnels.  I pulled everything out to make sure of that.

I got the memory usage of my main application way down since posting this (I did have a tunnel in there).  I was still seeing this extra usage that didn't make sense to me so I put the question out there.

[Ben]

Although reading the whole thing and dealing with as one big chunk of data is easier to code, the user simply can't see it all at once. I just finished up an enahncement to an app to allow the end user to "replay" the app do fast forward etc. THe challenge I was facing that the amount of data that was collected (even in raw form) was more than could fit into memory. I worked arounf that issue by only displaying the current state and left the rest of the data on disk. It ran with much less CPU and memory demands than I was anticipating.

Aside:

A "reverse" button was also implemented. I never saw a chart "un-plot" data before this project. down-right disturbing if you ask me.

If you want to stay with the monolithic approach then an AE may be able to help you. Create an action that reads the data and do your file read in that case (not a sub-VI!) and cache the binary data in a Shift Register. In another action, select the record on interest, forat for display and return same to the caller.

If coded right, you  data will sit one buffer (the SR) and only the data associated with "now" is required for the update.

Just trying to help,

Ben

[Jason_P]

The reason I asked was that I have a similar application where I store spectrometer snapshots (arrays of 2048 singles) in a binary file at a rate of 1-10 per second, sometimes over the course of several hours.  When I want to review the data, I have an XY graph that displays one snapshot at a time.  Underneath the graph, I have a slider that I use to index the file and pull up the desired snapshot.  I can quickly run the slider back and forth and watch how my spectrum evolves over time.  It runs smooth and uses next to no memory... I was thinking something similar might have worked for you.

Jason

[dwj314]

Ben-

AE?  Application Engineer? 

Thanks for the suggestions.  I have tossed around ideas for smarter loading and viewing of the data but haven't had a chance to try them out yet.  I squashed a pretty big memory issue yesterday that makes what I am seeing here look like nothing so I'm not too worried right now.  

-dennis.