Memory usage data to cluster

[bmann2000]

I've inherited an application that performs many functions on a Cluster of data and it would be a significant amount of work to redesign using a different data structure.  At the very start of the process, data is loaded from a tab separated text file and reshaped into a 2D array of double using Spreadsheet String to Array.

When I try to load a large file of 200MB, I get an out of memory error.  It's quite clear why this happens, I need to parse off the first line and first column of data prior to populating the cluster.  In doing so many copies of the 2D array are made and LabVIEW runs out of memory.

I'm familiar/aware of the inplace structure but cannot figure out how to redraw the diagram.  Should I be pre-allocating memory for the cluster? Please see attached.

[majoris]

In these situations, I normally use a Data Value Reference which can also be used with the in place structure. If you're going through the trouble of using the in place structure, might as well use the DVR (both are hassles).

[nathand]

What are you trying to show with this VI? The in-place element structure is only useful for saving memory if you're modifying existing data. In your case you're replacing the cluster elements entirely - the left-side terminals are unused and unwired - so you get no benefit.

You will get some benefit from moving all the front-panel terminals to the top level of the diagram, outside the case structure (find the "Clear as mud" thread on this forum for details; that discussion was probably over a decade ago, but the information as far as I know still applies).

[Ben]

You can find the Clear As Mud Thread here.

If you are reading the file as strings that kills a lot of memory.

You can also process the file (?not sure if you are using a file) line by line to decrease the memory demands.

Ben

[bmann2000]

I've attached more code this time to make it clearer.

I'm stuck with the format of the GridXY file, see exampleGridXYfile.gxy.tab.  This is how the data is supplied by those designing the 3D shape to be machined.

I'm hoping to retain the Cluster processMap because I have several VIs that operate on the Cluster, rotating arrays etc inside inplace structures.

The example file is 3.5MB, it will load fine.  My target file is 200MB.  It seems reasonable that LabVIEW could load 200MB from text file and populate a Cluster.

I modified the code to read line-at-a-time as suggested by Ben.

Suggestions please on where to go next.  If I do need to  implement Data Value Reference, will it allow me access array subsets without increasing the memory used, i.e. solve the problem outlined in the attached code.

[majoris]

I made a small attempt at cleaning up the issues. A few things:

1) Be careful about loading the full files. I changed the size function to show you how to get the X and Y size without loading the full file into memory as an example.

2) Be careful with loading 200MB stuff into the front panel. You occasionally had multiple indicators and controls that contained the full data. This made the VIs use 5 times the amount of memory!

3) I have used a DVR as an example. Access the data in the DVR using an in place structure, particularly when trying to copy the data out of the DVR. If you use the delete DVR function, it data will no longer be available using the reference.

Review what I added here. Hope this helps. (I saved this for 2013 since that seems ot be what you were using).

[nathand]

---

@majoris wrote:

3) I have used a DVR as an example. Access the data in the DVR using an in place structure, particularly when trying to copy the data out of the DVR. If you use the delete DVR function, it data will no longer be available using the reference.

---

The DVR doesn't gain you anything here. What are you hoping to accomplish by using one? I think you have a misunderstanding about the benefits of a DVR.

[majoris]

DVRs ensure serial access to a unique memory allocation which is useful when dealing with a large data blocks. It also prevents large data sets from getting placed in controls and indicators which can sometimes accidentally get saved in the VI which is a disaster. Essentially it lets you be a bit less careful about copying references to the data without praying the compiler always understands your intentions, specifically across VI boundaries. If you want to see what I achieved by using one, simply compare the memory performance before and after the changes I made. This change will reap large benefits as the VI hierarchy grows though.

[nathand]

---

@bmann2000 wrote:

I'm hoping to retain the Cluster processMap because I have several VIs that operate on the Cluster, rotating arrays etc inside inplace structures. ...

I modified the code to read line-at-a-time as suggested by Ben.

... If I do need to  implement Data Value Reference, will it allow me access array subsets without increasing the memory used, i.e. solve the problem outlined in the attached code.

---

No offense, but you've gone about this all wrong, and completely ignored my simple suggestion. Again, move all your controls and indicators to the top level of the VI, outside all case structures. As strange as that may sound, it can improve memory use.

Do you have the option to zip up the 200mb file and attach it? Since it's text, and all numbers, it should compress to much smaller.

I don't have time right now to go through and attempt to fix your code, but here are some more comments.

First, there is no reason to do the line-by-line read AFTER you've already loaded the entire file. Are you able to run the RO_LoadGridXYZ-ArraySize VI on the 200mb file? If so, then there's no reason to do the line-by-line read, since you can already successfully load the file in memory. If you can't run that VI, then the rest of your approach won't work. And, if you are going to read line-by-line, open the file once before you start the for loop, and close it once after the loop ends. This is a lot faster, plus you won't have to keep track of the position within the file.

I would avoid using array functions when you know you're dealing with a scalar value. Where you are using Split 1D Array with an index of one, index out the element you want instead. Delete from Array is also an option, since you want the remainder of the array as well - although this only applies to the line-by-line implementation and I think you should be able to do it with the whole 2D array.

You mentioned that you're rotating arrays within an in-place element structure... that's going to be a slow operation for a large array, and doesn't necessarily require an in-place element structure. Many LabVIEW operations happen in-place already, and conversely, wrapping an operation in an in-place element structure cannot force an operation to happen in-place if that operation requires a memory copy. It's a hint to the compiler, not magic.

[nathand]

---

@majoris wrote:

Essentially it lets you be a bit less careful about copying references to the data without praying the compiler always understands your intentions, specifically across VI boundaries.

---

It would be a lot better not to be a sloppy coder 😉

All of the things you mention are easily avoidable with proper LabVIEW coding techniques (edit: except serialized access for edit operations, but that isn't an issue in the code shown here), and the code stays a lot simpler without a DVR. I'm not saying there's no use for a DVR, but I don't see one here. It's not difficult to tell the compiler what you're doing: put the controls and indicators on the top level of the block diagram, mark your inputs as required, and do as much work as possible in-place. The first two are easy, and a DVR doesn't help you with the last one.