LV RT 55% Memory Utilization Step Change

[sgregor249]

I have a program coded in LV8.5 RT that runs on a PXI 8106 controller. The program is designed to run continuously forever. I use the attahced VI that was reterived off of NI's website to monitor memory consumption. I use the VI to log total memory, available memory, and percent used. I was told that calling this VI excessively can hog resources, so I only call this VI  every 1.25 seconds. After about 5 days of running the software as a build, log files show a step change in memory consumption from 15% to 70%. I suspected that maybe one of the memory sticks on the controller was failing, but the logged total memory does not change when the increase occurs, however the available memory does descrease as expected.

Also note that this problem is not reproducable, that is to say it does not always occur 5 days after startup, tt seems to just randomly occur every few months . Also note that the PXI controller is rebooted weekly. The software shuts down when memory increases this much to ensure safe operation of the system, and the customer resets the PXI controller and the memory consumption returns to normal. However, the perviously metioned course of action has non-desirable reliability and scheduling impacts.

I do not believe that any of the following are the cause of my problem, being that they would show a ramp increase in memory consumption over time:

          Filling an unbounded queue and not properly servicing it.

Creating references that are never closed in a loop.

Concatenating strings or building arrays in a loop.

Continuously obtaining a named notifier or queue in a loop.

If memory serves me correctly, I have observed this behavior on two separate PXI-8106 controllers.

Has anyone seen anything like this before? Does anyone have any ideas of what might be happening? Lastly, I'm not allowed to post the code.

Message Edited by sgregor249 on 02-16-2010 07:25 AM

[Ben]

I have seen similar.

Use VI Analyzer against that code and pay attention to the items listed under the performance checks.

Reader's Digest reply...

LV allocated contiguous block of memory for buffers. If your code does something that overflows the allocated block, it has to find another one that is larger (and at least years ago.. it was double the previous size). This can happen for the reason you mentioned or if a file grows needing a larger buffer or if an array of cluster (for example) gets a filed filled that was previously empty (preallocated array of error clusters that have empty source strings now have text in them).

Ben

[sgregor249]

I ran the VI Analyzer against the top level VI block diagram. The only performance issue was that debugging is enabled for the top level VI.

[Ben]

---

sgregor249 wrote:

I ran the VI Analyzer against the top level VI block diagram. The only performance issue was that debugging is enabled for the top level VI.

---

Puzzled.

Do you have reason not suspect issues in sub-VI's?

Note:

Without seeing your code my hand are going to be tied very soon, so you may want to review the links in this Tag Cloud instead of me playing multiple guess.

Doing what I can with the limited data available,

Ben

[sgregor249]

Heh good point ben. I assumed that doing the top level diagram would include all sub-vi diagrams. I'll try to find out how to include subVIs or I can write a program to get all subvi paths and call the vi analyzer on them.

[sgregor249]

I was able to run the VI Analyzer on all VIs. It did find some build arrays/concatenate strings in loops, but none of them are of concern. Ben, you say that you've ran into a similar situation before, do you know what version of LabVIEW you were using and if/how you resolved the issue?

[Ben]

---

sgregor249 wrote:
I was able to run the VI Analyzer on all VIs. It did find some build arrays/concatenate strings in loops, but none of them are of concern. Ben, you say that you've ran into a similar situation before, do you know what version of LabVIEW you were using and if/how you resolved the issue?

---

I have been involved in RT apps going back to LV 6i so I have seen a lot.

The solutions were driven by the problem. In some cases I changed how data was stored and accesed, it all depends on the nature of the problem. At this point I can offer the following ideas;

1) RT Trace Execution Tool kit can find a nd flag buffer allocations while running. You will have to force teh app into the failure scenaio to use this approach.

2) Careful analysis of log files and attempt to corelate the jump with code actions.

3) Pink cluster are dangerous. Brown are of fixed size while pink can change. Avoid pink when possible and don't trust the pink if you do use them. (Most of my fixes involved pink wires)

4) If you have a valid service contract with NI, send them your code and ask them to advise.

5) If you have a local LV user group, attend there next meeting and see if they can help.

6) Talk to your local Ni rep to see if they can help.

7) I can not recall any issue with the NI supplied software ever causing a memory leak.

😎 I avoid shared variables and the like so if you are using those, I can not rule out issues off-hand regarding buffering etc.

Ben

Shameless plug follows:

I have consulted on these types of issue in the past and helped people find and fix the issue. If you run out of options and have a budget, see my signature how to contract for my assistance.

End of Shameless plug.

[sgregor249]

Any other ideas?

[Ben]

I don't give up on helping easy.

Some old bench marks I perfromed showed that when a memory buffer is too small and needs to be expanded, LV will double the size of what it was.

So if you are seeing a 55% step then i would have to guess it was using about 25% before it had to grow. If the buffer was using 25% of your memory then it would probably (yes I doing some guessing! but I have to guess since I can't see the code) it is either

A) You largest single buffer in the app that is growing or

B) A brand new large buffer is created.

I don't know if that helps but that is all I can do for now.

Ben