TestStand 2020 64-bit resource or memory leak tracking

[Brett_Percy]

Hi all,

I’m working on a 64-bit TestStand 2020 system that uses a custom OI and the parallel process model to test between 6 and 18 DUTs in parallel.  The system makes extensive use of DQMH modules both in the OI and the driver level

We’re having issues with memory usage creeping up inexorably as the test sequence is repeatedly run, with result collection being a possible culprit after some initial investigation

We don’t want to start using on-the-fly reporting if it can be avoided, as there’s already concerns about testing time.  The test sequence is very short, on the order of 10-15 seconds.  There’s no instrumentation involved, just a serial-over-USB connection to the DUT for each test position

Has anyone got any useful tips for profiling TestStand to try and nail down where memory (or other resources) is being allocated and perhaps not being freed?  Turning parts of the system on and off and trying to use Task Manager or Resource Monitor to diagnose things is not really cutting it

Cheers

Brett

[drohacek]

Hi Brett - 

Have you seen the article Troubleshooting Memory Growth Issues in TestStand Systems? It provides both theoretical guidance and practical tools to help diagnose memory leaks in TestStand systems. NI is very interested in any feedback you have on this content. 

[Brett_Percy]

Thanks drohacek,

I've been looking at https://www.ni.com/en-au/support/documentation/supplemental/16/characterizing-memory-issues-in-tests... so far, I hadn't had a close look at this other doc but I will check it out.

So far I've tried disabling report generation and result capture as per this doc, and it looks like the problem is in the sequence rather than reporting as the memory usage still goes up.  In fact it seems to grow even when the sequence is not actively testing a part.

There's a loop in our PreUUT sequence that contains three Expression steps that check the state of a couple of flags to see if a DUT has been placed in the testing nest, plus a short Wait.  As far as I can see these expressions aren't doing anything that would be consuming much memory, but when the sequence is sitting spinning around this loop, it can increase memory usage anywhere from a few to tens of MB per minute.  This is in TestStand's Sequence Editor, I haven't checked this in our OI yet

I've paused all twelve executions of the sequence in the DUT polling loop, and this appears to prevent the memory usage from increasing.  

There is a DQMH module that's responsible for setting the flags that this loop is checking.  The module still runs even if the test sequence is paused, so it seems to be pointing to the sequence rather than the module as the source of the problem.

Disable Result Recording for All Steps is set for the PreUUT sequence, so TestStand shouldn't be capturing any values from the loop

Cheers

Brett

[richjoh]

I have never used the parallel model. I would use the model with 1 DUT, run your test or try to replicate the memory issue using 1 DUT.  You've already replicated with many DUT it appears and you've isolated the PreUUT sequence.  If there is only 1 DUT Parallel model,  PreUUT should not consume - is my guess.

[CyGa]

Hi Brett,

What these expressions are doing exaclty ?

Are you dealing with arrays in the TS variables ?

[Brett_Percy]

Hi Richjoh,

Thanks for the suggestion

I've just tried out running a single execution instead of the six I was testing with, and memory usage appears to be flat - which is interesting.  Still using the parallel process model.

One other slightly odd thing that happens when the memory usage climbs, is that the private memory for the SeqEdit process sometimes drops significantly, by half or more, but the commit charge will keep going up.  This is what eventually blows it up, when the overall commit charge for the system reaches 100% we start getting processes fail to start up, particularly when reporting runs

Cheers

Brett

[Brett_Percy]

Hi Cyril,

The loop is made up of

While(

AllOf(!FileGlobals.SocketFlags[RunState.TestSockets.MyIndex],
FileGlobals.ContinueTesting,
!StationGlobals.Exit),

)

Expression 1

Locals.TempString = Str(!FileGlobals.SocketFlags[RunState.TestSockets.MyIndex]) + " " + Str(FileGlobals.ContinueTesting) + " " + Str(!StationGlobals.Exit)

Expression 2

FileGlobals.ContinueTesting = !StationGlobals.Exit

Expression 3

Locals.Handler_Firmware_Rev = 0,
Locals.Handler_Serial_No = 0

End

So that's it, the only array reference is reading from the SocketFlags, which is where our DQMH module indicates it's time to start testing.  There's a helper VI that catches the broadcasts from the module and sets the SocketFlags

All of these steps are preloaded when the file is opened, but other than that nothing really unusual

I've tried running 12 instances of the DQMH module just in LabVIEW and they seem fine by themselves

Cheers

Brett

[CyGa]

Reading that last test with 1 vs 6 stickers makes me curious.

It's expected to consume more memory with 6 sockets as the processes are duplicated.

But I wouldn't expect a flat memory consumption in 1 configuration vs a rollercoaster memory consumption in the other configuration.

Seeing this makes me think that TS is dealing with allocation + deallocation processes.

Do you see the same behavior with the memory of you pause all/one the preUUT execution using 6 sockets ?

[Brett_Percy]

I've tried pausing all of the executions, and that stopped memory usage from rising.

I haven't tried pausing all but one yet, I'll give that a try and post the results

[CyGa]

Is the DQMH module filling the array thanks to the TS API ?

Just asking...but, to me, the tests you already are not really pointing in that direction...

However if the DQMH module deals 6x times more with the API and leaks, you might see a problem faster with 6 sockets...