Problem with upgrading Custom Step Types across Test Sequences

[Grefyne]

I have encountered a problem when making changes to a custom step type in the Type Palette, in that I need this change to be able to be propagated across multiple sequence files that contain this custom step type.

I have done a small test to get my problem to the simplest form. I created a simple .NET assembly that did basically nothing, then I created a custom step type in the Type Palette that used the above .NET assembly. I then created a test sequence that used this custom step type and saved it. At this point everything is fine, and the sequence executes as expected.

Then I closed the sequence, and moved the location of the .NET assembly. I edited the custom step in the Type Palette to reflect the new location of the .NET adapter. When I loaded the original test sequence, I was prompted on which definition of the step type I wanted to use. I picked the currently loaded type from the Type Palette. As far as I was concerned this should have updated the test sequence with the new step type.

But once I tried to run the test sequence that had been upgraded, it could not find the .NET assembly. Hence the sequence had not been upgraded to use the custom step type. While the above demonstrates one aspect of the sequence not getting updated ie the .NET assembly location - the same is true for other parameters.

Which leads to the question, why ask what version you want to use when a conflict arises, if it always seems to use the version from the test sequence and not what I specify when prompted?

My other question is, given that I want to make changes to a custom step type. How do I apply the new test type across potentially many sequences that have the old type embedded in them?

Maybe I have missed something, but to me this seems to be a major drawback?



I have seen what I believe the same problem mentioned before in the thread below, but never got a satisfactory answer to the problem above:

http://forums.ni.com/ni/board/message?board.id=330&message.id=6235&requireLogin=False

[AllenP]

Hi Grefyne,

There are a couple of things to keep in mind when using a Custom Step Type.

The first is that most of the settings that you are altering are default values. There are ways to force values to be shared between step types, but I recommend being a bit careful when doing this.

For example:

I will create a custom step type with a default module. In your case, you are choosing a .NET module. However, this is only a DEFAULT. For example, if you have two instances of this step, you could change the module to point to completely different code modules. In other words, TestStand only looks at the step type to determine what module it should use when first creating the step. There are a few ways to get around this problem.

The recommended way to use the behavior you are describing is to use a post-step instead of a default module. In this case, I would also disable the "Specify Module" under "Disable Properties". Post-steps, unlike code modules, are unique to step types. This means that EVERY instance of a loaded type must have the same substeps. If a type is loaded into memory that has a different substep, a type conflict will occur and you must choose which instance of the type you want to have loaded in memory.

There are a few other ways to solve this problem that are more useful for other properties. For instance, suppose I wanted to always use the same post-expression for a step type. The typical way to do this is to use a default expression, and disable the property so that the "customers" of your step type are unable to change it. But suppose you already had several instances of this step type in sequences. There are still ways to correct the problem. First, if you load all of the instances of this step type into memory, there is a checkbox at the bottom of the step type properties dialog that allows you to apply all changes to the loaded instances of a step type.

However, this is not always possible. If you are unable to load all of the sequence files and make the necessary changes, there is one other remaining solution. There is a "Shared" flag on every property. Whenever you set a property as "Shared", all of its attributes, including value, are shared into one instance. This can have some "dangerous" consequences that you should be aware of. First, if you change this property in ANY location, it will change the property in ALL locations (there is only one actual property, and it is shared between them all). An example of a property that I would NEVER recommend sharing is a result value. For example, if you have two numeric limit step types that perform two different tests. If the result.numeric property is shared, running either of these two tests will change the result.numeric value of the other. This means, you will get the same value for both tests in your report! Also, all flags (such as Include in Report) are shared by using the Shared flag.

Hopefully, depending on your situation, one of these workarounds will work for you. This topic can be a bit complex. If you have any additional questions, or something was not quite clear, let me know, and I will elaborate further.

[Grefyne]

Hi AllenP,

Well I tried your suggestion of using a post-expression and followed the exact same procedure. The only difference is that it did indeed update the sequence to reflect the new path to the DLL. It works, as I would have expected it to. But it has raised a few more questions for me:

1) You mention that the recommended way is to use a post-step. I was wondering where you found the recommendation; can you point to anywhere on the NI site or documentation that advises this? Or is this something that you have found out yourself?

2) At the end you mention that these are workarounds. Does this mean you'd consider TestStands current behaviour a bug?

3) How do you know what parts of a custom step type get upgraded and what parts don't?

Basically I need to ensure that the upgrade path for my custom step types will work seamlessly, but can't find much in the way of best practises or documentation. Do you know of any good reference examples?

As an aside, we have actually purchased some custom step types from a 3rd party that use the ActiveX/Com adapter. Looking at their calls, they use the adapter under the general tab and not a post-step. We are experiencing major problems upgrading our sequences when they release a new version of their custom step types - basically it doesn't work for reasons I believe to be much the same as the above. Does this mean they have gone about creating their custom step types in the wrong way? This is possible, as I would say that creating the custom step types isn't part of their core activities.

Many thanks for your help so far,
Ian

[RayFarmer]

Hi Ian,

I drawback I have found using the the Post-Step method, during developement you can't step-in the code of the Post-Step as you can with the code module. You have to have initial setup your breakpoints prior to executing your code.


I would be interested in the hearing what the latest thoughts are and adding my two cents worth on your question:

For question 1)

I haven't found any of the current Teststand documentation that makes this recommendation.
There are a couple of application notes, Style Guide and Creating Configurable Test Step, but both of documents make no recommendation. Infact the Creating Cofigurable Test Step document still uses the Code Module method.


There is also some tutorials with example code, "Custom Step Types - A Tutorial using the CVI Stand Adapter Prototype" and a similar one using labview that use the Post-Step mechanism rather than the code module. Again they make no recommendation to use this route.

A third document I came cross was called "How Do I Make a Custom Step Type?" but this used the code module method.

Your can find this information by doing a search on the Entire Site for "Post-Step".



For question 2)

I came across this problem when Teststand 3.0 (beta testing) come on the scene, and the extra faclities that was provided with this version. I did make some suggested recommendations but I was given the cold shoulder.


For question 3)

The TestStand Reference Manual sort of hints at what gets changed. But you would probably only really know by experiencing the problems, similar to youreslf.


Regards
Ray Farmer

[AllenP]

Ray is mostly correct in his responses. I just wanted to post a bit of additional information.

1) I should have posted in my previous post, but I am on the TestStand R&D Team. If you look at any of our step types that always call the same "module", you will notice we use post-step's to do this. Also notice that we disable the ability to select a code modules. Examples of these steps are the Message Popup step and the Call Executable step. By design, a code module is designed to be customizable per instance. Use of the "Shared" flag is a relatively new feature, so it has been more of an exploit than the designed behavior.

2) This is designed behavior. A step type is only a type, not an explicit definition. Perhaps workaround was the wrong term. What I meant is, if you would like to use this architecture without redesigning all of your sequences, there are a few tricks to do that. I wouldn't want you to have to redesign an entire architecture to fix this one problem, although I would recommend a different design if you were starting from scratch.

3) The easiest way to determine if something gets passed on to step instances or remains at the type level is to examine the properties associated with each step. Make sure you have hidden properties enabled so that you can see them all. You will notice that substeps are not included in an instance of a step.

The custom steps you purchased probably should have been designed with post-steps rather than step modules. On the other hand, the same method of sharing the code module will probably be the easiest way to fix the problem.

This issue is one that is very easy to mistake. I think that we could explain this better in the future.

Allen P
TestStand R&D