NI TestStand Idea Exchange

Post an idea

Currently, you have to choose between these options on each file (and the only indication of the choice is in the sequence file properties dialog). Both options can be quite useful in the Parallel and Batch Model (of course there isn't much of a difference in the Sequential Model). Clunky workarounds also exist like using Shared File Globals and having arrays for anything per-execution or using Separate File Globals while putting anything shared in Station Globals or in LabVIEW Functional Globals, but I think native support for both in the same file would be much cleaner and easier.

This could be just having two "FileGlobals" categories in the Variables pane (I prefer this), or it could be a right-click option on each individual File Global.

View All (2)

TestStand UserManagement Update

Provide a UserManagementCallbacks.seq to replace the currently hard-coded built in TestStand User management so that we can implement our own site-specific user management.

Sequences should allow for

1) Retrieving and returning of a user container from a database or file of our choosing (login)

2) Storage of new user credentials to a database or file of our choosing

3) Adding a New User with the option to reject (i.e. we don't like the name "!#%ThisGuysName!!!")

3) Deleting a User with option to reject deletion (i.e. we don't want to allow deleting user "Administrator" or "ThisReallyImportantUser"

4) Updating User Information

5) Logout, return a default user or shutdown on logout.

6) Retrieving of All Users available for login.

7) Check if a UserExists

This would be in addition to the FrontEndCallbacks that handle the Login/LogOut process.

View All (1)

Array Literal Support for Named Types

I would like to ask to add named types support for TestStand array literals. The current behavior is described as follows:

Declares a one-dimensional array of numbers, strings, Boolean values, object references, or containers. If all elements are of the same type, the result array is an array of elements of that type. If all elements are not of the same type, the result array is an array of containers.

I would like it to be more or less like this:

Declares a one-dimensional array of numbers, strings, Boolean values, object references, containers, or named type. If all elements are of the same type, the result array is an array of elements of that type. If all elements are not of the same type, the result array is an array of containers.

The problem with the current implementation can be seen on the screen.

It is not a bug (BUG 1828580 to be more precise), it is a feature.

Empty Array Literal

Currently, there is no unambiguous solution for passing empty Array of X (where X is a different type than 64-bit Floating Point) as a sequence parameter. If we use {} it is assumed to be of type 64-bit Floating Point. To have an empty Array of Strings we can use ambigues Split("",""). I have no idea how to pass e.g. empty Array of Signed 64-bit Integer without creating empty variables.

It would useful to have Empty Array Literals of a particular type e.g.:

{} - default

{}i64 - for Signed 64-bit Integer

{}ui64 - for Unsigned 64-bit Integer

{}s - for String

{}b - for Boolean

{}r - for Object Reference

{}c - for Container

{}TYPE_NAME - for type definition where TYPE_NAME is type definition name; e.g. Path.

The concept of TestStand, a sequence. Sequences of groups. Sequences of Sequences.

Pre Sequence, Post Sequence, Pre Test, Post Test. Run as a process, iterates on product instance. Single Up, Multi Up. Is incredibly flexible.

The paradigm is for test. No reason to constrain it to test processes. The paradigm cannot be adapted for an process that runs sequences, and iterates.

Test Stand can be used as a vehicle to build and manage pipelines for Machine Learning Processing. For both training and inference.

For example - I want to do some ML training with hyperparameter tuning. Queue up the variants and run the sequences, compiling the results.

Better Yet, Andrew NG is pushing data centric AI. Use test stand to queue up data variants, train and eval a model. Auto generate the results.

The BIG Tech platforms all support data pipelines.

TestStand has it built in.

Looking at any step result, there's a TS.StartTime property, but it's referenced against the "the number of seconds since the application initialized the TestStand Engine"

What this means is that each step result is referenced against a somewhat arbitrary and changing epoch that isn't recognizable outside any one instance of the TestStand application, let alone by external applications.

I'd like to propose adding TS.StartTimeUnix, which contains the absolute time the step started.

I've added this in our model customization plugins, but to convert the epoch to UNIX I have to perform this calculation:

Seconds(False)-Seconds(True) + Parameters.Result->TS.StartTime

Why I'm asking for this:

In our step result database table, we've added a column for STEP_START_DATE_TIME which is absolute UTC time. To populate this I have to go through the gymnastics above. I revisited this code today and thought, "What silliness. This should be something natively supported." (Other than that I have no strong opinion on the matter)

View All (7)

Smart rename into/out of container

(This idea was created as response to this discussion board topic: https://forums.ni.com/t5/NI-TestStand/ExpandPathMacros-FindFile/m-p/3767542#M56811)

The commands "Engine.FindFile", "Engine.FindPath" and "Engine.ExpandPathMacros" only support one macro:

$(Platform)

but there are other generic paths that could be helpful e.g. when trying to call a third-party .dll (that's my main usecase) or write a logfile:

-------

$(commonprogramfiles) - The actual folder name varies depending on Windows language settings and bitness. No absolute paths possible but a lot third-party .dlls are here.

$(temp)

$(localappdata)

$(appdata)

$(user)

$(public)

$(MyData) - The actual folder name varies depending on current user and Windows language settings

It hopefully shouldn't be too hard to access the windows functions which get the actual names via macros.

The concept of TestStand Environments was introduced in TestStand 2016 and allows you to define multiple configurations on a single test station or development PC. It would be nice if there was an option to link a sequence file to a specific environment, similar to how you can set a sequence file to "Require Specific Model" In the Advanced tab of the Sequence File Properties. When a sequence file is loaded, the engine could check to see if the required environment matches the current environment and:

Automatically relaunch the app in the correct environment
Generate an error
Prompt the user for an action
Other??

It would be nice to choose the behavior when a required environment doesn't match.

If anyone has other thoughts or use cases, please share below.

-Trent

View All (3)

The Search Directories.Insert method should only insert the directory if it is not already there.

The Method includes an index argument, if the directory is already there, then it should move the existing directory to the requested index.

While we were working on the shipping examples for DQMH, we discovered that the insert method was creating duplicates every time it was called. We implemented a work around that includes a for loop to check each of the items in the search directories list to see if it is the directory we are trying to insert, if it is, we delete it. Once the for loop ends, then we insert the directory where we want it.

You can see a video of the issue and how we worked around it here: DQMH 3.1 Only inserts the Delacor examples directory into Search Directories once

Please make possible to select LV Development System version in similar way as it is possible to select Run-Time Engine.

It also should be available in TS API.

That will allow Test Engineers to use code modules (especially those inside .lvlibps) from different versions.

It would be also useful for to set up desired version of LV for code modules without lack of debugging options of Run-Time Engine calls.

TestStand Expression Macros

Let's say I'm building a fairly long compound expression that has some repeated parameters - I would like a way to specify (without creating additional locals) a variable/macro for use just within that expression.

For example, instead of:

Locals.CommandLine = "cmd /c C:\some path\that\is\reused"
Locals.CommandLine += " /path C:\some path\that\is\reused"

I could specify:

#pathmacro "C:\some path\that\is\reused"
Locals.CommandLine = "cmd /c " + pathmacro
Locals.CommandLine += " /path " + pathmacro

The idea being that with this I only need to update the macros in one place in a long expression.

This is a simple example, but hopefully you can see why this would be useful and why I wouldn't want to create lots of locals when the values are required only within the expression.

View All (2)

Modify sequence during excution

There should be an Engine Callback that is executed as the very last step in the order of step execution that will execute regardless of any and all settings or step results: the PostStep "No-Matter-What" Engine Callback. I have a requirement to perform certain actions at the end of every client sequence step regardless of step, station, report settings as well as step results (Status, Error). Right now, the callback that gives the best coverage is ProcessModelPostResultListEntry, but this does not fire when client sequence developers set Result Recording Option for a step to False. My requirements call for my actions to execute whether or not the developer of the client sequence desires the results to land in the TestStand report or not. As with other callbacks, if it is blank, the engine can skip it.

The PostStepNoMatterWhat callback would execute regardless of all these, but as an aid to Framework developers, NI should provide a matrix for each of the default Process Models that shows which of the engine callbacks will execute given the following data&colon;

Step.Result.Status {Done, Skipped, Passed, Failed}
Result Recording Option {Enabled, Disabled}
Step.Result.Error.Occurred {True, False}
Run Mode {Normal, Skipped}
Ignore Run-Time Errors {Enabled, Disabled}
On-The-Fly Reporting {Enabled, Disabled}
Error Dialog Selection {Ignore, Run Clean-up, Abort}

TestStand API SearchDirectory objects in TestStand have the following writeable properties:

Disabled
ExcludeFileExtension
ExcludeHiddenSubdirectories
FileExtensionRestrictions
Path
SearchSubdirectories

When you use the SearchDirectories.Insert method to add a new search directory, you can only specify the following for the new search directory:

Disabled

ExcludeFileExtension

FileExtensionRestrictions

Path

SearchSubdirectories

It would be really nice if the SearchDirectories.Insert method also had an additional input parameter for setting the ExcludeHiddenSubdirectories property for the new search directory. This would result in a more consistent interface and fewer API calls when working with new search directories.

Because of the way .NET applications and assemblies are invoked in TestStand they are a child process of TestStand. This means that they share TestStand's resources. For most applications this is not an issue but if the application or library being instrumented by TestStand is resource intensive this creates a significant problem. In the scenario that served as the impetus for this suggestion we saw performance 1/10 that when running the target application outside of TestStand.

To correct this I recommend the .NET adapter architecture be changed or be able to be configured such that instead of directly instantiating target applications a call to create an object with a .NET adapter would create a separate process that consisted of a TestStand WCF client wrapper process that would host the target .NET process and communicate with the parent TestStand instance via WCF.

Here is a simple block diagram of the intended architecture:

Let consider I have an array of containers. The container is defined as a user type, (array of the same elements) and consist, let say, from three string variables A, B and C. And, let say the array size is for, example 4.

Now, let say, the developer needs to access the array of elements A. Let say he/she has to create an array od strings called StrArr So what he has to do is for example like below:

FileGlobals.StrArr={FileGlobals.MyContainerArray[0].A, FileGlobals.MyContainerArray[1].A,FileGlobals.MyContainerArray[2].A, FileGlobals.MyContainerArray[3].A}

It is a lot of writing isn't it?

Wouldn't be worth considering to implement implicit conversion which allow just to write

FileGlobals.StrArr=FileGlobals.MyContainerArray.A

FileGlobals.StrArr=FileGlobals.MyContainerArray[].A

View All (4)

NI TestStand Idea Exchange

Add some Maths functions for array: mean, variance, standard deviation...

Have Both Separate File Globals Per Execution and Shared File Globals in the Same File

TestStand UserManagement Update

Review all idea Exchange and Update status

Array Literal Support for Named Types

Empty Array Literal

TestStand Sequences for Machine Learning Pipelines

For each StepResult, add StartTime with UNIX time offset

Smart rename into/out of container

Additional macros for "Engine.FindFile", "Engine.FindPath" and "Engine.ExpandPathMacros"

Provide ability to link a Sequence File to a specific Environment File

Allow model plugins access to Engine callbacks

Search Directories Insert method should not duplicate a search directory

Programatically allow to select LV Adapter Development System version

TestStand Expression Macros

Modify sequence during excution

Add New PostStepNoMatterWhat Engine Callback

SearchDirectories.Insert method should have additional input parameter for excluding hidden subdirectories

Host .NET processes outside of TestStand's application space

Implicit and fast container array elements access.