How do I extract the Test Results from a TestStand 4.0 execution in a C-Sharp Application?

[tlaford]

I am writing a C-Sharp application to initiate a test sequence in TestStand 4.0.1 and then collect the Test Results to update our own screen(s) / Report(s).  I will need some type of recursive routine to collect the following information (if it is available) for each step: Name, Result, Min Value, Max Value, Measured Value, Units.

My attempts have been to use an End Execution Event to collect the data:

    void EndExecution(object sender, EndExecutionEvent ev)
    {
        PropertyObject results = ev.exec.ResultObject.GetPropertyObject("ResultList", 0);

        int numContainers = Results.GetNumElements();

        for (int i = 0; i < numContainers; ++i)
        {
            PropertyObject resultData = results.GetPropertyObjectByOffset(i, 0);
            string testResult = resultData.GetValString("Status", 0).ToString();

            PropertyObject testStep = resultData.GetPropertyObject("TS", 0);
            string testName = testStep.GetValString("StepName", 0).ToString();

            MessageBox.Show(i.ToString() + ". " + testName + " = " + testResult);
        }
    }

But this only gives me the MainSequence data.

How do I traverse the ResultsList tree to access all the test steps' results?

[dug9000]

The ResultList is a recursive data structure. If the step's results you are looking at is a sequence call, then its result will have a result list for the sequence it calls under its TS.SequenceCall.ResultList property.

Hope this helps,

-Doug

[tlaford]

Here is what I came up with for my first pass It seems to work, but I'm not sure how bullet proof it is.

void EndExecutionEventHandler(object sender, EndExecutionEvent ev)
{
    try
    {
        PropertyObject report = ev.exec.ResultObject.GetPropertyObject("ResultList", 0);
                
        // Loop through all the objects in this hierarchy
        int numElements = report.GetNumElements();
        for (int i = 0; i < numElements; ++i)
        {
            // Retrieve each object and parse it's property hierarchy
            PropertyObject resultData = report.GetPropertyObjectByOffset(i, 0);
            WalkTree(resultData);
        }
    }
    catch (Exception ex)
    {
        MessageBox.Show(ex.ToString(), "End Execution Event Handler");
    }
}

private void WalkTree(PropertyObject item)
{
    // Determine if this property needs to be stored
    populateResultData(item);
    // If this is a ResultList, access each array entry
    if (item.Name == "ResultList")
    {
        // Determine how many array entries exist
        string lowerBound, upperBound;
        int numElements;
        PropertyValueTypes elementType;
        item.GetDimensions("", 0, out lowerBound, out upperBound, out numElements, out elementType);

        // Loop through all array entries
        for (int k = 0; k < numElements; ++k)
        {
            // Retrieve each array entry and parse it's property hierarchy
            PropertyObject arrayEntry = (PropertyObject)item.GetValVariantByOffset(k, 0);
            WalkTree(arrayEntry);
        }
    }

    // Loop through all sub-properties
    int numSubProp = item.GetNumSubProperties("");
    for (int j = 0; j < numSubProp; ++j)
    {
        // Retrieve each sub-property and parse it's property hierarchy
        PropertyObject subItem = item.GetNthSubProperty("", j, 0);
        WalkTree(subItem);
    }
}

private void populateResultData(PropertyObject item)
{
    switch (item.Name)
    {
        // If this is a StepName property, extract and store the value
        case "StepName":
        {
            string dataValue = item.GetValString("", 0);
            // Now use the string
            break;
        }

        // If this is a Status property, extract and store the value
        case "Status":
        {
            string dataValue = item.GetValString("", 0);
            // Now use the string
            break;
        }

        // If this is a Low (Minimum Value) property, extract and store the value
        case "Low":
        {
            string dataValue = item.GetValString("", 0);
            // Now use the string
            break;
        }

        // If this is a High (Maximum Value)  property, extract and store the value
        case "High":
        {
            string dataValue = item.GetValString("", 0);
            // Now use the string
            break;
        }

        // If this is a Numeric (Measured Value) property, extract and store the value
        case "Numeric":
        {
            string dataValue = item.GetValString("", 0);
            // Now use the string
            break;
        }

        // If this is a Units property, extract and store the value
        case "Units":
        {
            string dataValue = item.GetValString("", 0);
            // Now use the string
            break;
        }

        // If this is an Error Property, extract and store the error code and message
        case "Error":
        {
            if (item.GetValBoolean("Occurred", 0))
            {
                string errorCode = item.GetValNumber("Code", 0).ToString();
                string errorText = item.GetValString("Msg", 0);
                // Now use the string
            }
            break;
        }
    }
}

Is there a better way to do this? Comments welcome.

[Snowman 6008]

Tla,

I would recommend that you take a look at the report generation sequences: C:\Program Files\National Instruments\TestStand 4.2\Components\Models\TestStandModels  reportgen_xml.seq would be a good place to start, although they all rely on navigating the ResultList. Just currious what are you doing with the ResultList once you have it?

[tlaford]

Richard S,

Once I have the string values of the fields I am interested in, I use them to update a custom report generated by my C# application.

T

[RayFarmer]

Does seem a bit strange why you are doing this because TestStand will do this for you!

Why do you feel you have to resort to this instead of using the Teststand report generation.

Regards

Ray Farmer

[tlaford]

Because that's my job.

We have a Csharp application that was written to manage TestStudio executions and report on the test execution results.  I have to modify this homemade application so it will also work with TestStand and produce the same reports.

[dug9000 · Accepted solution]

tlaford,

There are a couple of big issues with what you are doing:

1) It's better to call your result processing code from a TestReport callback sequence (just use an action step) or modify the process model to call your code rather than calling it from the UI's EndExecutionEventHandler, because if you use Test UUTs then there is a separate resultlist per UUT, not per execution, and it is generally processed at the end of each uut not at the end of the exection. If you use the EndExecutionEventHandler then you should NOT be using the process models that ship with TestStand because they do not function this way. You should instead just run your sequences without a process model. Though I really recommend going the process model route, even if you write your own custom one, because it's more flexible then hardcoding your result processing into the UI like you are doing. Some reasons why a process model is more flexible/better are that it will work in any UI including the sequence editor (no changes to the UI are needed), and it can do anything you can do in a sequence, for example call a code module in any supported language, call subsequences, asynchronus sequences, etc.

2) Recursing through every property at such a low-level is bad because the names of properties aren't guaranteed to be unique to what you are expecting them to be. Anyone can write their own custom step types with whatever result properties they want to have. You need to make sure the full path to the properties is what you expect it to be, and you might even want to check that the step type is the one you expect. You should instead loop on the top-level result items ONLY and look for the full lookupstrings of the data you are expecting from such items. For example, rather than WalkTree you might have the following instead:

// This should be called from a step in a TestReport callback or in a custom process model
// rather than from the UI's EndExecutionEventHandler, because if you use Test UUTs then
// there is a separate resultlist per UUT, not per execution, and it is generally processed
// at the end of each uut not at the end of the exection.
void ProcessResultList(PropertyObject resultList)
{
	int numElements = resultList.GetNumElements();
	for (int i = 0; i < numElements; i++)
	{
		PropertyObject item = resultList.GetPropertyObjectByOffset(i, 0);
		string stepTypeName = string.Empty;

		if (item.Exists("TS.StepType", 0))
			stepTypeName = GetValString("TS.StepType", 0);

		if (stepTypeName == "NumericLimitTest")
		{
			if (item.Exists("Limits.High", 0))
			{
				 double highLimit = item.GetValNumber("Limits.High", 0);
				 // Do something with the value
			}
		}

		// NOTE: Both SequenceCall step type and NI_Wait step type can have subresults like this. A wait step that waits on an asychronous thread or execution will have this.
		if (item.Exists("TS.SequenceCall.ResultList", 0))
		{
			// Recurse for sequence call results.
			PropertyObject sequenceCallResultList = item.GetPropertyObject("TS.SequenceCall.ResultList", 0);
			ProcessResultList(sequenceCallResultList);
		}
	}
}

Hope this helps clarify things,

-Doug

Message Edited by dug9000 on 04-28-2010 10:09 AM

[tlaford]

Doug9000, 

I am a TestStand novice, so please forgive my ignorance.

1) I don't know what you mean by a "TestReport callback sequence".  Is it just a different Csharp event I can register for?

I can not change the SequenceFile that is being executed.  Once a SEQ file is reviewed and approved, it can never change without going through the review process again.  So adding a data collection step to the file isn't an option.  But, once the SEQ file is loaded (and before it is executed) I could make calls in my Csharp application to insert a data collection step as the last step.

If modifying the default process model is the correct solution, can it be modified via calls inside my Csharp application, or is it a change that would need to be done before hand using the Sequence Editor?  What would the change be (via code or Sequence Editor) to call one of my Csharp methods to collect the data?

2) Your sample code looks like it would be more bullet proof.  Is there a list of which properties are valid for each StepType?  That list would allow me to expand your code snippet to extract all the data items I need.

[tlaford]

Additional thoughts:   Let's forget the EndExecution event handler for a moment.   I have a Csharp method that is called when the execution is completed, is there some way to access TestStand at that point to get to the data I need?