NI TestStand Idea Exchange

Top Authors

User

Kudos

Post an idea

There should be an Engine Callback that is executed as the very last step in the order of step execution that will execute regardless of any and all settings or step results: the PostStep "No-Matter-What" Engine Callback. I have a requirement to perform certain actions at the end of every client sequence step regardless of step, station, report settings as well as step results (Status, Error). Right now, the callback that gives the best coverage is ProcessModelPostResultListEntry, but this does not fire when client sequence developers set Result Recording Option for a step to False. My requirements call for my actions to execute whether or not the developer of the client sequence desires the results to land in the TestStand report or not. As with other callbacks, if it is blank, the engine can skip it.

The PostStepNoMatterWhat callback would execute regardless of all these, but as an aid to Framework developers, NI should provide a matrix for each of the default Process Models that shows which of the engine callbacks will execute given the following data&colon;

Step.Result.Status {Done, Skipped, Passed, Failed}
Result Recording Option {Enabled, Disabled}
Step.Result.Error.Occurred {True, False}
Run Mode {Normal, Skipped}
Ignore Run-Time Errors {Enabled, Disabled}
On-The-Fly Reporting {Enabled, Disabled}
Error Dialog Selection {Ignore, Run Clean-up, Abort}

View All (4)

Let consider I have an array of containers. The container is defined as a user type, (array of the same elements) and consist, let say, from three string variables A, B and C. And, let say the array size is for, example 4.

Now, let say, the developer needs to access the array of elements A. Let say he/she has to create an array od strings called StrArr So what he has to do is for example like below:

FileGlobals.StrArr={FileGlobals.MyContainerArray[0].A, FileGlobals.MyContainerArray[1].A,FileGlobals.MyContainerArray[2].A, FileGlobals.MyContainerArray[3].A}

It is a lot of writing isn't it?

Wouldn't be worth considering to implement implicit conversion which allow just to write

FileGlobals.StrArr=FileGlobals.MyContainerArray.A

FileGlobals.StrArr=FileGlobals.MyContainerArray[].A

View All (4)

When developing custom TS sequence analyzer methods, it would be beneficial to be able to control how the message gets reported. Specifically sometimes I would like to automatically add the message into the "ignored" list.

Reasoning -- I have determined that there is a problem with the code, but based on other stuff I think the user really intended and needs the code written this way, so I still want to flag that I found it, but have it in the ignores list so that the user won't by default see the reported message, but if they go to the ignored list they can see it.

Example: If I were doing the #NoValidation directive from scratch, I would have written the rule such that if the "expression validates correctly" rule would check the expression validity regardless of any #NoValidation directives. Then it would report the message (assuming it has a validation error). If the expression contained #NoValidation it would be automatically put into the "ignore" list. This way we have the ability to see that the expression did not validate properly, but recognize that the user has included a flag in their code to specify "this is OK and I want to ignore it" so it goes into the ignored list. -- All done automatically by the analyzer code module without the user having to manually click on the ignore menu item in the Analysis Results window.

View All (3)

It looks like there is no "gentle" way to access the results of the executed tests in ongoing execution during this execution.

Sometimes there is a need to access the test results during the execution, before the data will be committed to the database, and execution is still ongoing. The reason for that could be we can reuse the some data of the test in other tests, or we can use for example the status of the test to drive the flow in our sequence.

It looks like there is no other general way to do that as only described by Sasha here: http://forums.ni.com/t5/NI-TestStand/RunState-ProcessModelClient-How-to-access-the-results-from-the/td-p/2796658

So, theoretically, - please read Sasha post - we have recipe to access all results we want. However, problem with accessing the result list is that, that it is done via the index of the ResultList array.

It leads us to two problems:

1. the elements in that list depends on the step position in the sequence file, which makes the editing sequence almost impossible,
2. if our sequence contain loops the problem from the point above is even more impossible.

Therefore, the idea:

Please prepare the easy accessible, not index based as it is now, method (container?) which developers can access the Results containers of the steps on the fly during the execution.

Handler proposal 1:
Step name (binded as unique ID) + execution order number

Handler proposal 2:
Callers path + StepName +execution order number

where execution order number could be the handler which could be number 0 by default unless the step is called few times.

Other post: https://forums.ni.com/t5/NI-TestStand/Accessing-measured-data-from-further-steps/td-p/1144710

View All (4)

This is a carryover from an earlier idea that got halfway implemented (when the close "x" got moved to the tab).

Right now in TS2025 out of the box sequence editor, if you have a file name that is really long, it just gets truncated in the tab. This makes sense if you have lots of tabs open and are going to be competing for space, but if there are only one or only a few files open, there is lots of extra space that could be used to clarify the file name.

Request is: if you have extra space in the UI for tab sizes to get bigger, make the tabs large enough to display the full file name without truncation.

The Test header detail for 'Failure Chain' hasn't really kept up with modern test data scope being asked for of test headers.

Having full access to step details around failure, as part of failure chain in header, would be wonderful, but in our constrained world at a minimum, including both step name, and step Requirement ID (if any logged as part of report) would be a big advantage, and would avoid the need for some very complex report customization (digging through the recursive logic around failure-chain assembly) on behalf of TS architect community whom are being asked to tease this type of information up to header by their quality team stakeholders.

Ideally, some mechanism per step-attribute, perhaps bundled into additional parameter for "Execution.includeAdditionalResult()" API call, where we can add flag for 'also include in failure header' would be hugely valuable so that developers can not only add data to body of the report, but also cherry pick a subset of that added data that is reported even into failure chain in event of failure.

other step attributes that would be high-value in a failure chain might include if step is a looped step, what index was the reason for the failure, if step looped N times, or step's unique execution index (if step name is reused many times across test sequence as part of sub sequence calls or other reuse of code)

Currently my quality team members can see failure/error via quick query in DB for header info, but then have to do some very hairy SQL logic in order to find the database entry for the specific step result that failed during that execution, and fetch the requirement id(s)-as-flattened-string stored for that failure.

View All (3)

This may be a combo of LV and TS ideas exchanges but it affects TS mostly from my current experience. Issue occurred in LV 2022 called from TS2023. Hopefully already fixed but I know it also existed in LV2016 and TS 2016.

The issue is explained nicely in this forum post.

https://forums.ni.com/t5/LabVIEW/Packed-library-contains-folder-structure/td-p/3587329

Suggestion is to ignore the OS based heirarchy when selecting a VI in TS and to only use the virtual folder/library structure for selection of VIs to use. This will avoid all sorts of issues of paths changing on rebuild which we've just spent hours debugging.

Hi there.

This may seem like a radical idea, and probably my most blunt idea I've ever posted but please bear with me:

NI folks working on TestStand as software, I don't mean to be offensive. I understand being in the position of having to make incremental changes, eventually outgrowing the original design of the software. I'm just saying what I think needs to be said and is long overdue.

Anyone who has had to customize the NI Report Generator will know that it is a complete hodgepodge of incremental changes from over the years.

As a high level summary:

Within NI_ReportGenerator.seq, all of the report types (ATML, XML, HTML, text, etc.) are handled in the same sequences
Depending on the report type, different steps are executed via preconditions, Post Actions (essentially GOTOs ☠️) and somewhat rare if-then flow logic.
This tight coupling of the report types makes it really difficult to follow the flow of steps that execute, and it's also just bad software design.
Further, there are sequence calls set by expression depending on the report type, making it even harder to follow. I'm guessing that this was supposed to be extensible by adding more ReportGen_<type>.seq files, but we're long, long past that being realistic.
Nomenclature is often inconsistent
Naming of steps often doesn't match what's really happening.
Preconditions and post action conditions are very often implicit.

AllOf( This, that, AnyOf( Sometimes the other, or another still), ! The other options that might apply)

&& !( Other implicit conditions because post actions in prior steps precluded execution of this step )

Highly technical details follow:

First off, in the TestReport callback, if report generation is disabled, the message to display is stored in Locals.ReportSection.AsReportSection.Body. Tricky, tricky, you didn't fool me! 😄 Yet, why not use an appropriately named variable?

Also, in the configuration dialog, the report style is referred to as the "Report Format", but the report format in the code is independent of the report style. The nomenclature isn't consistent. Along this theme, in the c code, there is a struct for ReportOptions, but it's not the report options from the TestStand sequence, and it's gradually populated over several consecutive function calls.

Taking ATML, for example, depending on which ATML standard is selected, the execution flow varies substantially, is inconsistent, counterintuitive, and seemingly arbitrary:

ATML 2.02

In the TestReport callback, if and only if the ATML standard is 2.02, the step "Get Report based on ReportStyle" is called.
- ...but it's only for one style, and not even a style, which would be "xml"
- This Get<report type>Report (GetATMLReport) sequence only exists for ATML (no other formats) and is only called for ATML 2.02, thereby completely removing any justification for a sequence call by expression.
GetAtmlReport then calls one of two functions in ATMLSupport.dll, depending on whether the caller has initialized a report section
The report section is always initialized two calls up in "Model Plugin - UUT Done", so there is one case with a DLL call that never executes and is dead code.
Ultimately, the Get_Atml_ReportSection function is called in ATMLSupport.dll.
In ATML_Report.c, this calls Get_Atml_Report_Impl, which is where most of the business of appending report strings takes place

ATML 5.00

In the TestReport callback:

Locals.ReportGenSequenceFileName is determined by looking for the first four characters of FileGlobals.ATML202StandardReportStyle. (i.e. ATML) This is regardless of the ATML version, even though there are respective constants for each ATML version. You might incorrectly assume that it only applies to ATML 2.02
This time, by precondition we skip execution of "Get Report based on ReportStyle", which is a misnomer and exclusively for ATML 2.02
Counterintuitively, we also skip "Get Report Body (DLL)", even though we've configured body generation to be via DLL. By precondition, this step is implicitly for any format other than "xml", which I guess means ASCII and HTML? But they're not explicitly named.
No, we execute "Get Report Body (Sequence)" instead, even thought we're configured to use a DLL. Via various preconditions and post actions, this step is implicitly: Not for ATML 2.02, not for ASCII, and not for HTML. So I guess that means ATML 5.00, ATML 6.01 and XML?
"Get Report Body (Sequence)" is yet another call by expression to an external reportgen_<format>.seq
"Get Report Body (Sequence)", when called for these XML based report styles, does not use a sequence. In all of those cases, the AddReportBody sequence it calls calls a DLL and only has half a dozen steps, so it's not a sequence, really.
Within AddReportBody, there are two possible DLL calls based on whether Parameters.ReportSection is null, but it's never null because the calling sequence always initializes Parameters.ReportSection. (More dead code)

I thought about talking about ATML 6.01, and I could go on, and on... and on. I think you all get the idea.

This is an absolute nightmare to navigate and very messy to customize without an extreme makeover crying edition.

Surely we can do better.

Thank you for reading my missive.

Mr. Jim

View All (4)

Enable Zoom IN in the Label Description field

View All (3)

Hi NI Teststand developers

Don’t you find it hard setting up Teststand with all the different settings possible, scattered in al the different locations?

Why is it not possible to put all this settings in one form with tabs. (like in MS VisualStudio)

Especially the Cfg directory is packet with possible stumbling rocks.
This improvement includes the configuration menu.

I usually spend a lot of time just setting up TS. When finally arriving in the deployment state. I stumble again trough this setting forgetting important things because they are scattered throughout Teststand.

It would be an immense improvement of the TS usability

If you start improving this, don’t forget to adjust the editable field sizes of forms etc., so they can be read and edited in a practical way.

Another thing I find not really user friendly, is the form settings lock. When I try to load a workspace and the lock is set (worked in TS2019), I get an error. Why this?

It would be nice to hear your statement to this, and when you find time to improve this.
If you need more improvement hints (there are some more) , don’t hesitate to contact me.

My team uses Analyzer Rules to enforce best practices for developers across sequence files and workspaces with great results. However we would love to be able to also provide rule checking and warnings for developers on some basic *.tsd file content such as Installer Name field, and build-path & installation-path/image-path, etc... New (or rusty!) developers often miss steps in their configuration and needlessly struggle with badly behaving deployments.

As TSD files do not appear to have an API available from TestStand at this time nor conform to a json/xml syntax internally, for easy parsing via 3rd party tools, some method(s) for 'reading' file (or converting to readable syntax) may also be required in order to build rules effectively?

TestStand Container Parameter

Hi,

it can be a simple and stupid question, but I don’t get why, when I define a container as a parameter of a sequence called in the main sequence. That parameter can’ be expanded to fill the different field with the variables that I need. I created also custom type variable for trying to achieve this aim but nothing. I thing that passing a container as parameter can be helpful in particular when you need to pass a lot of variables regarding the same “object".

Best Regards,

Zuc

Download All

View All (3)

View All (1)

If I call a python function within the TestStand Python adapter the arguments for the functions must be according the order of the function which isn't Python like. This means the order of the arguments in TestStand must be exactly the same as in the Python function, the name of the argument is ignored.
It would be really nice if the TestStand Python adapter would support Keyword Arguments as well.

https://www.scaler.com/topics/python/types-of-function-arguments-in-python/

View All (2)

NI TestStand Idea Exchange

Add New PostStepNoMatterWhat Engine Callback

Improvement request: In the Call sequence step please make the list of sub-sequences to call sortable.

Implicit and fast container array elements access.

automatically add TS analyzer messages to ignore list

Variable multiassociation directly from the module output

Make the access to the test results data during the execution more easy.

Make FileName Tabs Larger in Sequence Editor

Include Requirement ID data of the step as part of Failure Chain in Test Report Header

LV Packed Libraries to use only project hierarchy for VI selection in TS

Rewrite / Refactor NI_ReportGenerator.seq (NI Report Generator) and its Myriad Callees

Enable Zoom IN in the Label Description field

Add some Maths functions for array: mean, variance, standard deviation...

Improve Drag-and-Drop Behavior in Sequence Editor

Auto Generation of FileGlobals/Locals

Disconnect from Custom Type using "right click" like in LabVIEW Typedef

Consolidate all Teststand settings in one form & different improvents

Ability to write Analyzer Rules around content of TestStand Deployment Files (*.tsd)

TestStand Container Parameter

Deployment Utility : LabVIEW Packaged Library Version Locked to Deployment Version

Python adapter supporting keyword arguments