Adding new tests at runtime?

[jeffmccracken]

I am considering using TestStand but need to do the following ; Develope a test sequence or suite that will test a base UUT configuration and deploy this as a test application. The UUT configuration can vary hardware wise. It will sometimes consist of just the base configuration and other times it will take on different forms, but alway contain the base configuration.  I would like to then develope other deployable TestStand tests, that depending on the UUT configuration could at run time be added to suppliment the base configuration tests or could at least be executed as well under programatic control of the original TestStand application. 

In LabWindows I can do this pretty easily with a Plug In type architecture. The base application would support the base UUT tests. If it is discovered at runtime that the UUT is base config plus optional components, the base application will load one of several possible supplemental test DLL's that will programatically add new tests to the base application, inserting them into the test sequence. It can also programatically modify the base apllication UI.

So I guess the question is : Can a deployed TestStand application modify it's test sequence with additional new tests at run time. The new tests would be unknown at the time the TestStand application is built and deployed.

Thanks alot

JeffM

[James_Grey]

I haven't tried it but I strongly suspect that modifying a sequence while it is running is not supported and would lead to errors or crashes.

However, you can dynamically create new sequences or sequence files at runtime, so tests created at runtime could be run in a subsequence.

You can also specify the target of a sequence call step by expression, so that the actual subsequence called varies at runtime.

Even simpler, you could use preconditions or If steps to skip calls to subsequences that test components which are not present.

I'm not sure exactly how you want the UI to change dynamically, but steps in your sequence can display dialogs or they can send custom UI messages to your customized operator interface application. The application could change its UI depending on the messages it receives. If the application must support UIs that are not known before runtime, the message could specify a panel that the application places in a plug-in hosting area it defines. At this point, the problem doesn't have much to do with TestStand and is more of an issue of how to componentize UIs in the development environment you use. TestStand would just be relaying a UI request from a step module to the test application.

[Odd_Modem]

Use the sequence concept to do what you want. Your top level sequence might be a sequence that calls 'N' sub-sequences. Out of the gate, there are only one or two sub-sequences that actually do anything, and all the others just return from call. Later on, fill in the the other sequence calls with the stuff you want. These are sort of like call-backs, eh? Pre-conditions are a very powerful tool, and to make the most of them, you need to delve into the sequence properties, and TestStand properties, to Access the conditions of execution.

[daijoubu]

Jeff, I like the idea of using a sub-sequence that uses an expression to load a sequence file.  You would have a main sequence that gets deployed to test the base unit.  In this main sequence you have an expression that creates an array of sequence file names based on the options on the UUT.  Then create a loop around a Call Sequence File step.  This call sequence file is set to dynamically load a file.  I have attached a demo of this method.

To implement this, you would need to come up with a standard for option sequence file names so your main sequence would be able to find them.  A property file with an array of structures that describe each sequence file might work, but this would need to be updated with every new option.  Even better would be a step that can index the sequences in a specific directory and create this array of structures dynamically.  The structure would contain the filename and the option that it tests.

This is something that I put together in about 10 minutes this morning.  The report looks a little ugly...  If you implement any of the above ideas or have improvements please let me know.  This is an interesting set up. 

[daijoubu]

This idea has been floating in my head all week so I wrote a more complete article to walk you through how it was created.  I plan to add a couple of more articles that expand this idea by dynamically creating the list of sequence files to run and cleaning up the report format.  Take a look at the article and leave some feedback.

[RayFarmer]

Hi,

You could use a custom step type based on the SequenceCall steptype, which has additional properties such as:

Step.SequenceName
Step.SeqFilePath

Then you can uses the PropertyLoader to load these direct.

You would setup your steptype so that the 'FilePath or Reference' control to used the property Step.SeqFilePath and the 'Sequence' control to use Step.SequenceName.

This way you are not restricted to just calling the sequence "MainSequence".

Regards

Ray Farmer

[daijoubu]

I plan on doing a second part to this article that covers finding the files.  I had considered using a property file to hold a list of all the optional sequence files (plug ins) but decided not to.  The property file would introduce too many complications for people developing the plug ins.  Who owns the property file if you have more than one developer creating plug ins?  What prevents an old plug in from overwriting the property file and deleting the information about a new plug in?

I am considering using a property file for each plug in to hold metadata about the plug in such as what sequence to run, who the developer is, what features it supports, etc. 

My goal with these articles is to create a true plug in architecture.  A set up where the original developer does not need to know about any future plug ins.  These future plug ins could be developed by anybody.  I'm curious how far I can push TestStand to accomplish this.  I realize this might be straying a little off topic from Jeff's original post but refrained from starting a new thread as I haven't seen any threads promoting articles like this in the boards here.

[jeffmccracken]

I just want to thank those that have responded and the effort involved. I will take some time to try and digest all of this input. I really know nothing about TestStand other than what I can read on the NI site so it may take me some time to figure this stuff out. Again I appreciate the responses.

JeffM

[jeffmccracken]

daijoubu

The idea of a plugin architecture is not off topic at all as this is really what I'd like to be able to do. I just didn't think to frame the question in that way.

JeffM