Out of memory on saving values to indexing tunnel

[VitekStepien]

Hi,

I have a large while loop that processes a lot of data, but up every iteration only puts 2 values from two different arrays into indexing tunnels, to save after the loop has finished execution. At some point, LabVIEW stops and pops up and out-of-memory error, at about 720,400 samples or so.  

I know for a fact, that the size of the file that should be written isn't very large, at 500,000 samples the file takes about 15 Mb on disk so nothing serious.  Also, before I used to save whole 2D arrays of the same data, instead of saving 1D array of last samples, and the output files would be >500 Mb but would still save.

I feel like there's something about the length of the array exceeding LabVIEW capacity, but I can't pinpoint it...  I would appreciate any help. See a snapshot of the code below.

[GerdW]

Hi Vitek,

---

@VitekStepien wrote:

every iteration only puts 2 values from two different arrays into indexing tunnels, to save after the loop has finished execution. At some point, LabVIEW stops and pops up and out-of-memory error, at about 720,400 samples or so.  

See a snapshot of the code below.

---

I don't think this small part of your code is enough for us to pinpoint your "out of memory" problems.

• What about attach the whole VI/code?
• What about attaching a sample VI with similar code, that shows the same problem?
• What about putting both elements into a 1D array before wiring to just one conditional tunnel? (Both tunnels use the same condition?)
• What about the upstream data? There are two 1D arrays (of unknown size) where you only use exactly one element (first/last)!? Why collect arrays at all?

[BertMcMahan]

I very much doubt it's the array. I'd bet you have a While loop that's not terminating like you think it is. Unfortunately we can't debug a tiny picture of a subset of code. Can you post the whole thing?

[VitekStepien]

The whole vi is large, but I'll try to give you an idea of what's happening. The vi. basically solves a set of coupled differential equations over some length (done inside the "RH4 amp" vi in the center of the top left quadrant), and every iteration of the loop is a step in time - that's why I need to operate on 1D arrays.  I'm mostly interested in the last value at every iteration, that's why I only store 1 value for each iteration into the tunnel. I'm running it with the Boolean "Full data?" set to false, so the 3 tunnels which gather 1D data into 2D output are completely empty the whole time.  If I'm not saving the data but just running the vi looking at the outputs, it can run "forever" without errors.

Here's the error message I get:

[Bob_Schor]

... and don't post (only) a picture -- be sure to include the actual VI (a file with the extension ".vi") to allow us to "drill down" (or "scroll across") to see all that we think is relevant to your problem ...

Bob Schor

[crossrulz]

With the note saying "To data saving module", it makes me think you should be doing the logging in a separate loop so you don't have to grow the array in the first place.  Have a good look at the Producer/Consumer.

[altenbach]

So you are extracting "first" and "last", discarding the rest. wouldn't it make more sense to just keep two elements in a shift register (1D array with two elements), update whenever they change, and forget all these conditional tunnels.

Yes, we need to see real code. A picture is useless.

[VitekStepien]

GerdW, Bert McMahan, BobSchor, unfortunately I can't post the whole thing, it's proprietary...  Are there any specific things you think I could try, or situations you know of that could case this kind of problem?  I would assume you can help me debug it even without seeing the code yourself.  I tried running the vi memory diagnostics, but it landed me nowhere. 

crossrulz, I know the architecture and I thought about that, but the problem is I would either have to append a new datapoint / batch of datapoints continuously to a file, or build an array in the consumer loop which is kind of counterproductive.  Also, I use MATLAB script node to save it directly to a mat file which I need for further data processing, which makes appending the file multiple times during runtime even more complicated (although I will look into it).

altenbach, I need the first and last elements from each loop iteration, so I don't see how a shift register could work.  For n-loop iterations, I actually need to save n- first and lasts points. 

[JÞB]

From glancing at a picture I can see that your array wires are branched multiple times and then fed into subvis that likely have not been inlined. YOU HAVE MORE COPIES of the data than you think.

I'll generate a new LabVIEW Proverb.  "Spaghetti belongs on a plate."

Consider the following:
•Create a DVR with concurrent read access.
•Pass the DVR to inlined subvis everywhere you operate on the data.  And that should include passing the DVRs to subvis to the m scripts. 
•Grow the arrays in the DVRs in an IPE rather than on SRs and mark it a stomper. That way when the array grows and needs to reallocate a larger block of memory, only one reallocation is needed rather than one reallocation for each data copy!  

You might even consider passing the DVR via NOTIFIER of CHANNEL WIRE to the various consumers of your array data.

[altenbach]

You seem to be randomly blaming the conditional tunnel while the entire VI is a gigantic hairball and the problem could be anywhere.

I a not going to play "Where's Waldo", but from a casual glance, I can't even find the conditional terminal of the while loop in question. Where is it? What's the mechanical action of the various buttons? What's happening in the various subVIs? Why are there so many local variables? What's up with all the Matlab script nodes? (You are not running pure LabVIEW, so external code could be contributing to the issues!). What is the memory use over time when looking at the task manager?

Can you reduce the code to a simple simulation with one while loop simulating the various 1D arrays, then tell us what you want to keep once the loop stops.