pause and resume teststand

[bilby42]

Hi,

I've been using Teststand (V4.1), Labview (V8.5), and switch executive (2.1) recently to design several production testers (using a modified version of the batch process model), and I'm confident with the general design of "normal" multi threaded, multiple UUT test system. However I'm now required to design a burnin/stress test system that has a "unique" feature. I would greatly appreciate some advice from this group to ensure my architectural decisions are sound.

As a background, the burnin/stress test rig will comprise several sockets which will all run in parallel. Each socket could potentially have a different type of UUT connected. Each socket will also have a small thermal chamber associated with it that will enclose the UUT during the test. Instrumentation such as an switches (multiplexer and matrix), AC source, DMM and pressure calibrator will be shared between sockets using switch executive routing and normal Teststand locking mechanisms. Each UUT test will be quite long in duration (hours). All the UUTs are direct off mains, i.e. they have live 320vdc buses!!!

Unfortunately the burnin/stress tester has a unique requirement... from time to time operators may want to open a chamber door for inspection before the test is completed. Because of the live nature of the UUTs and the fact that instrumentation is shared via switching across all chambers, the chamber doors will have interlock switches fitted that will drop ALL the AC supply outputs to ALL UUTs on any chamber door openings. This is required because I cannot guarantee the status of the switching matix and any associated rogue cross routes from any adjacent live UUT chamber. The interlock will use an industry standard safety switch/relay system (probably a Guardmaster), to meet regulatory safety requirements.

Hence all the UUT test sequence threads need to be placed in a "pause" mode at this stage, and when the chamber is closed again must resume tests from where they left off (this is the part I'm having the most problem with).


Hence to the bit I need critique on.....

I'm using the parallel process model and overriding the "SequenceFilePreStep" callback, which is called for each sequence step for each socket. By inserting an API query on the "RunState.CallingStep.UniqueStepID" in this callback and monitoring the chamber door interlock switches, I can determine the exact location that each UUT test sequence was at before the operator "paused" the tests. When a pause is detected I'll go into some sort of holding state, awaiting a "resume" command from the operator (maybe just monitor the chamber doors for a reclosure).

The hard part and bit I need advice on, is how to behave when a "resume" is detected. Obviously I need to reinstate the system parameters I cleared previously, i.e. I have to rearm the AC source, repower the UUTs, reset any UUT registers etc.

At present I've thought of two approaches....
1/ Jump back to the "setup" part of the last running subsequence. The SequenceFilePreStep callback will have a routine to keep track of the last entered subsequence for each thread so I know which subsequence to return to. Then its just a matter of forcing a jump to the setup part. However I'm not sure how I get Teststand to jump back to a predefined place for each socket. Assuming I've got the UniqueStepID for the relevant setup area, what API calls do I use to make this jump?

2/ My alternative method would be to jump back to exactly where each thread left off. In this case I would need to keep track of all my system parameters so I could call a specialised routine to reinstate everything without recalling the setup steps.

My gut feeling tells me "2" is the right approach, but it means I need to duplicate the setup routines each time (once in the actual subsequence setup area and again in my special reinstatement routine).

Is there a right and wrong way to do this? Am I using a totally incorrect approach, i.e. should I be trying to utilise a Teststand breakpoint command instead of my custom "pause" mechanism?

Note that my requirement isn't really a breakpoint due to the specific reinstatement steps before I resume. Also the operator will never be single stepping during pause as the UUTs will be effectively powered down and dead.

Sorry this post is so long. I hope someone can either improve on my approach or give comments on similar systems they have seen work in the past (help Ray Farmer 🙂 )

Regards,

Tom 

[bilby42]

I've been experimenting with two approaches to my pause/resume problem posted previously.

Firstly I've found its very difficult to place the UUT back into its original dynamic condition after the operator's "pause" operation due to the loss of power to the UUT (and subsequent loss of all the UUT's internal volatile registers). Attempting to keep a FIFO cache of the UUT's internal register structure gets messy and problematic. Even jumping back to the currently running subsequence "setup group" sometimes isn't enough, as a previously running subsequence may have configured some necessary UUT setting.

Secondly while I can play around with Runstate.Caller.NextStep to jump around in my sequence after a "resume", it makes for very hard to read sequences, i.e. the program logic is not obviously apparent to someone doing future sequence maintenance (especially because the "invisible" process model is modifying the sequence flow).


I'm now thinking of a third more intuitive concept thus.......

A normal burnin sequence has various delays and loops which are required to give the required burnin periods of hours. Running the burnin sequence without these delays/loops enabled would cause the burnin test to cycle rapidly through all its subsequences, effectively shortening the burnin from hours to minutes.

Hence if a normal burnin test is under way and the operator opens a chamber door, the "pause" mode is entered (door opening is automatically detected in the SequenceFilePreStep callback). This pause mode effectively terminates all UUT threads in the current execution. However before it does this it remembers where the "pause" occurred (i.e. what UniqueStepID was running at the time), and the fact that the test was interrupted.

When the operator recloses the chamber door, he is given the option of restarting from scratch or "resuming". Restarting from scratch simply runs the entire burnin test as normal. However "resuming" enables a flag that I'll insert in the precondition statement of any delay or loop step. If this "resume" flag is true the delay and loop steps will be skipped and the burnin test, while still starting from the beginning will tend to rapidly cycle through to the previously interrupted step location.

Once this interrupted step location is reached I can reset my "resume" flag back to false, causing all delay and loop steps to run as normal (i.e. the burnin test will slow to its normal rate). I can determine when I reach the interrupted step position by monitoring the UniqueStepId in the SequenceFilePreStep callback for each thread.

This approach guarantees the UUT is correctly setup after a resume and gives a more intuitive feel to the sequence flow for future maintenance, and is probably the way I'll go.

I would still appreciate any comments on better approaches.

Tom

[RayFarmer]

Hello Tom,

You seemed to have moved on from your original ideas since I looked at you post yesterday. I briefly looked at this and wondered how you could Pause the execution, without affecting the process that you might have running which is doing the monitoring and restarting process. But I see that you are not pausing as you do when you press the Pause button on the Operator Interface. 

Your latest solution seems a lot least complicated and hence will be easier to maintain.

One concern, is the fact that when you start up again you are running full speed through the burn cycle upto the point where you stop and hence maybe over stressing the UUT.

Regards

Ray Farmer

[bilby42]

Hi Ray,

Thanks for reading through my longwinded concepts. Re overstressing while running burnin at full speed.... It shouldn't be an issue as I'm applying same stresses as before. If anything it will be less stressful as I'll be applying each stress for a shorter period (until I reach my "pause" point). I may have to watch for transient disturbances on the UUT however, i.e. if I connect and then dump a load too soon after....

My current "issue" is that the HTML test reports (I'm using only report generation here), for each UUT will be broken into two pieces, a report showing results from the test start to the "pause", and a second report from the "pause" to the test end. I haven't yet thought of a way to stitch these two reports together automatically.

Thanks for the reply,
Tom