NI TestStand Idea Exchange

Post an idea

Hi there.

This may seem like a radical idea, and probably my most blunt idea I've ever posted but please bear with me:

NI folks working on TestStand as software, I don't mean to be offensive. I understand being in the position of having to make incremental changes, eventually outgrowing the original design of the software. I'm just saying what I think needs to be said and is long overdue.

Anyone who has had to customize the NI Report Generator will know that it is a complete hodgepodge of incremental changes from over the years.

As a high level summary:

Within NI_ReportGenerator.seq, all of the report types (ATML, XML, HTML, text, etc.) are handled in the same sequences
Depending on the report type, different steps are executed via preconditions, Post Actions (essentially GOTOs ☠️) and somewhat rare if-then flow logic.
This tight coupling of the report types makes it really difficult to follow the flow of steps that execute, and it's also just bad software design.
Further, there are sequence calls set by expression depending on the report type, making it even harder to follow. I'm guessing that this was supposed to be extensible by adding more ReportGen_<type>.seq files, but we're long, long past that being realistic.
Nomenclature is often inconsistent
Naming of steps often doesn't match what's really happening.
Preconditions and post action conditions are very often implicit.

AllOf( This, that, AnyOf( Sometimes the other, or another still), ! The other options that might apply)

&& !( Other implicit conditions because post actions in prior steps precluded execution of this step )

Highly technical details follow:

First off, in the TestReport callback, if report generation is disabled, the message to display is stored in Locals.ReportSection.AsReportSection.Body. Tricky, tricky, you didn't fool me! 😄 Yet, why not use an appropriately named variable?

Also, in the configuration dialog, the report style is referred to as the "Report Format", but the report format in the code is independent of the report style. The nomenclature isn't consistent. Along this theme, in the c code, there is a struct for ReportOptions, but it's not the report options from the TestStand sequence, and it's gradually populated over several consecutive function calls.

Taking ATML, for example, depending on which ATML standard is selected, the execution flow varies substantially, is inconsistent, counterintuitive, and seemingly arbitrary:

ATML 2.02

In the TestReport callback, if and only if the ATML standard is 2.02, the step "Get Report based on ReportStyle" is called.
- ...but it's only for one style, and not even a style, which would be "xml"
- This Get<report type>Report (GetATMLReport) sequence only exists for ATML (no other formats) and is only called for ATML 2.02, thereby completely removing any justification for a sequence call by expression.
GetAtmlReport then calls one of two functions in ATMLSupport.dll, depending on whether the caller has initialized a report section
The report section is always initialized two calls up in "Model Plugin - UUT Done", so there is one case with a DLL call that never executes and is dead code.
Ultimately, the Get_Atml_ReportSection function is called in ATMLSupport.dll.
In ATML_Report.c, this calls Get_Atml_Report_Impl, which is where most of the business of appending report strings takes place

ATML 5.00

In the TestReport callback:

Locals.ReportGenSequenceFileName is determined by looking for the first four characters of FileGlobals.ATML202StandardReportStyle. (i.e. ATML) This is regardless of the ATML version, even though there are respective constants for each ATML version. You might incorrectly assume that it only applies to ATML 2.02
This time, by precondition we skip execution of "Get Report based on ReportStyle", which is a misnomer and exclusively for ATML 2.02
Counterintuitively, we also skip "Get Report Body (DLL)", even though we've configured body generation to be via DLL. By precondition, this step is implicitly for any format other than "xml", which I guess means ASCII and HTML? But they're not explicitly named.
No, we execute "Get Report Body (Sequence)" instead, even thought we're configured to use a DLL. Via various preconditions and post actions, this step is implicitly: Not for ATML 2.02, not for ASCII, and not for HTML. So I guess that means ATML 5.00, ATML 6.01 and XML?
"Get Report Body (Sequence)" is yet another call by expression to an external reportgen_<format>.seq
"Get Report Body (Sequence)", when called for these XML based report styles, does not use a sequence. In all of those cases, the AddReportBody sequence it calls calls a DLL and only has half a dozen steps, so it's not a sequence, really.
Within AddReportBody, there are two possible DLL calls based on whether Parameters.ReportSection is null, but it's never null because the calling sequence always initializes Parameters.ReportSection. (More dead code)

I thought about talking about ATML 6.01, and I could go on, and on... and on. I think you all get the idea.

This is an absolute nightmare to navigate and very messy to customize without an extreme makeover crying edition.

Surely we can do better.

Thank you for reading my missive.

Mr. Jim

View All (4)

When user opens the Offline Processing Utility and at the same time starts to type on the keyboard the user can accidently rename the profile.

(Attached a screen-recording to visualize)

It would be great if there was no selected row or column when starting/opening ORPU. The renaming of the profile disturbs the production since the database logging will not work as expected.

When we start ORPU we already have the '/tray' enabled but somehow its still possible to accidently rename the Profile.

TestStand File Diff and Merge Utility is not very useful for code reviews on its own. It seems adequate for notifying the user that a sequence was added, however from the tool itself the user cannot actually review the newly added or removed sequence's contents. Why is there no + on the item tree to go deeper.

Alternatively:

If I have to right-click a sequence and select "Go to location" then why bother with the separate tool to begin with? Why isn't the diff utility integrated into TestStand's sequence editor itself? Seems like a side-by-side comparison within Sequence Editor would allow a reviewer to poke and prod around all the hidden settings that are often missed using the existing utility.

View All (1)

View All (5)

Looking at any step result, there's a TS.StartTime property, but it's referenced against the "the number of seconds since the application initialized the TestStand Engine"

What this means is that each step result is referenced against a somewhat arbitrary and changing epoch that isn't recognizable outside any one instance of the TestStand application, let alone by external applications.

I'd like to propose adding TS.StartTimeUnix, which contains the absolute time the step started.

I've added this in our model customization plugins, but to convert the epoch to UNIX I have to perform this calculation:

Seconds(False)-Seconds(True) + Parameters.Result->TS.StartTime

Why I'm asking for this:

In our step result database table, we've added a column for STEP_START_DATE_TIME which is absolute UTC time. To populate this I have to go through the gymnastics above. I revisited this code today and thought, "What silliness. This should be something natively supported." (Other than that I have no strong opinion on the matter)

View All (7)

Logarithmic Graphs in TestStand Reports

View All (5)

Load Schemas Programmatically

Hello,

It would be nice if there were an easy means of programmatically loading a database schema for result processing.

Here's my context for asking:

We have a custom database schema that must be loaded on all of our stations and we'd prefer that it "just works" when our customized database result processing plugin is loaded.

I managed to accomplish this by studying the format of TestStandDatabaseSchemas.ini and writing a parser to insert our schema as necessary. This was a bit painful, though, and the possibility of the format changing without notice has crossed my mind.

Thanks as usual to the R&D team. (Assuming this idea gets some traction! )

Kind regards,

Mr. Jim

View All (5)

Problem:

I create a model plugin and add some model callbacks to it. Then I want to access those model callbacks from my client file. I have to now add them to the process model so they show up in the list when I go to add callbacks. I get that I can add it as a blue sequence and it will work However, that is confusing to other developers if they don't know the plugin has the callback in it. It is painful to change the process model every time I create a new plugin because the process model could be used on hundreds of machines whereas my plugin may only be needed for 1 or 2 machines.

Solution:

Option 1: Show all callbacks from any plugins in the callback dialog list

Option 2: create a configurable list of model callbacks. Basically add any sequence files marked as model to the list and it will show all the callbacks from the sequence files in the list to the client, unless it cannot find the file in any search directory. Then it wouldn't show.

Option 3: From the add callback dialog allow the user to browse to a sequence file which contains model callbacks

I think adding this functionality will greatly help in making the plugins more abstracted from the process models. Just saying.

Cheers,

Hello,

As we can read in this very good document, we can add information in header for UUT in Report.

It is mentionned that we can do that for StationInfo in the same way:

"The report below includes the custom data in the AdditionalData container. The process for adding custom Station data is similar, but uses the Parameters.ModelData.StationInfo property instead of Parameters.UUT."

In fact, it doesn't work in TestStand 2014 and later, certainly the same for 2013 because the default report plug-ins doesn't support it as you can see in the help of TestStand 2016.

Maybe NI can add it like it is done for UUT.AdditionalData. The goal is to avoid to put some Station Info in UUT result to show it easily in reports.

Best regards.

MaximeR

CLA - CTD

The "Report Options" dialog box provides a lot of flexibilty in the way reports are generated for sequences executing under the Batch model. A new report can be generated for each UUT, for each socket, etc. One option that appears to be lacking, is to flat out not generate a Batch Report. Doing a brief search, I found at two other folks who were trying to do the same thing:

http://forums.ni.com/t5/NI-TestStand/Disable-Batch-Report-TestStand-2014/td-p/3091476

http://forums.ni.com/t5/NI-TestStand/How-do-I-disable-batch-report-in-the-batch-model/td-p/238387

Suggest adding another check box to the Report File Pathname on the Report Options dialog box to disable batch reports.

View All (1)

When running teststand deployment licence there is no Teststand GUI available to set or configure Results Processing.

I think it should be easy to export the active configuration to file. And then have a property loader type step to load the settings back into the sequence at runtime on the deployed machine. Note in my case I want to run different sequences with different reporting requirements on the same deployment PC.

I have found https://decibel.ni.com/content/docs/DOC-32076
and after more stuffing around than was justified I have got it working with TestStand 2014.

Also it should be much easier to specify a relative UDL file.

I resolved this by adding an expression after ReadEx that was:

Parameters.ModelPlugin.PluginSpecific.Options.ConnectionString="\"FILE NAME="+Left(RunState.SequenceFile.Path, Find(RunState.SequenceFile.Path, "\\", 0, True, True)) +"\\60467Database.udl\""

This means I can select an absolute path to the UDL file in the development directory on my laptop and then use a relative to sequence path in the deployment.

I feel there should be an more obvious way or am I missing something?

The suggestion is to support logging a datetime with millisecond accuracy to an sql database with an NI_DataOperation step.

I tried to implement this, but the milliseconds are not written to the database.

It would be very usefull to support this feature.

The datetime field in the sql database is defined as follows:

CREATE TABLE

dbo.UUT_RESULTS (

UUT_STARTTIME datetime NOT NULL

)

The datetime is logged to the database, but only up to the seconds, although the format string is defined as "mm-dd-yyyy hh:ii:ss.sss" and the string does contain the milliseconds e.g. "2-10-2016 11:49:15.505".

Software Details: TestStand version 2010 SP1, Operating System: Windows 7

There are some macros to specify report file name by expression in report options.

Specifying Report File Paths by Expression - NI TestStand 2014 Help - National Instruments
https://www.ni.com/docs/en-US/bundle/teststand/page/specify-report-file-path-by-expression.html

$(FileDate) contains speces in the string, e.g., "2015 12 15". If you want to delete space, you need to use $(FileYear), $(FileMonth), and $(FileDate) instead.

But there is no similar expression for time, i. e., there is only $(FileTime) and we do not have any macro such as $(FileHour).

SearchAndReplace function doesn't work for the macros. It is very helpful if we have $(FileHour), $(FileMin), and $(FileSec) as standard macros.

Before:

ReportOptions 01.png

After:

ReportOptions 02.png

We in our company are done now several test systems where the best process model is Batch Model. Customers however does not want other reports than UUT report from each test socket. Simple skipping of batch report generation is not possible at the moment particularly now when plug-in structure has arrive for joy of us all. For me this wanted feature is late now because I manage to find out solution, with a great support of local NI team. Finally this feature of easy batch report disabling would serve future generations and today newbies. They will save hundres of hours when they don't have to use try&fail -method to skip those few meaningful steps but just make one selection in report options.

It would be nice to be able to log all requirement links during execution in the report. I think the option should be similar to "Include Attributes".

Currently, loging requirement links in steps is easy as additional result (preconfigured, just include it in step settings).

But logging requirement links in sequences or sequence files is not easily done. It requires either a dummy step which transfers the requirement list to the step requirements or an overelaborate expression in the sequence call step.

I think a drop down would be a good option:

"Log requirements list:

- For all steps

- For steps and sequences

- For steps, sequences and sequence files"

I am not sure if more options are necessary as empty requirement lists wouldn't appear in the report at all.

Norbert

NI TestStand Idea Exchange

Rewrite / Refactor NI_ReportGenerator.seq (NI Report Generator) and its Myriad Callees

No selection of any field inside Offline Processing Utility (ORPU) when application starts

Review all idea Exchange and Update status

Integrate Diff Merge Utility into Sequence Editor

Axis labels for graphs in report

For each StepResult, add StartTime with UNIX time offset

Logarithmic Graphs in TestStand Reports

Load Schemas Programmatically

Create a configurable Model Callback list

Allow model plugins access to Engine callbacks

Support Additional Station Info into header for standard Report generation plug-ins

Provide mechanism to disable batch report through "Report Options" dialog box

Result Recording Option: Disable in Sequence Call

Export Result Processing Active Configuration

More Options in Include Step Results for Report Options

Support milliseconds logging to SQL database column with datetime format.

Using time string in report file path without space character

Easier way to skip Batch Report Generation in Batch Process Model

Option to log all requirement links in report

Able to select multiple rows while using 'Move Selected Results Up/Down' in the Additional Results settings