Test Report Generation Multiple Reports per Section of Test Sequence

[Oli_Wachno]

Hi,

I'm wondering if anyone has implemented this:

As a TestDeveloper, I need have a seperate Test Report for parts of my TestProgram in order to perform dedicated post-processing.

So let's say my TestProgram has three section and I want each one of them to create their own TestReport file.

Right now, I am looking into executing each section in an own execution using a ProcessModel. For sure possible, but it seems a bit complicated.

Anything you can point me to? 

Kinda not seeing the forest for (christmas) trees at the moment 😲

[bienieck]

I think that the easiest way would be to have each section in a separate sequence, called in a new execution.

The report object is part of the execution object so each execution will have a separate report.

However, if you want to avoid separate executions you can use Reports (collection of execution report objects). I think you can use API to create by yourself seperete reports and keep them in this collection. Unfortunately, this will require also results processing plugin tweaking. I think it is doable but it will require quite some time to figure everything out.

[j_dodek]

Hi Oli,

maybe the answer in this thread helps.
https://forums.ni.com/t5/NI-TestStand/I-want-to-generate-multiple-reports-in-Test-Stand/td-p/3778942

Best regards

Juergen

[Oli_Wachno]

Michal, Juergen,

thanks for your feedback!

@Michal: actually, I was considering a solution like you described, yet the resulting complexity was holding me back from trying

@Juergen: nice approach, just need to check which implications will arise. 

🙂

[bienieck]

Oli, I just realized that you could use multiple different Result Filtering Expression with multiple instances of the Report plugin. The filtering expression use Result property (from the result collection). Remember that it is a Result, not a Step.Result or anything else (see screenshot). So, you must determine which result property would be best in your case. I used Attributes (I'm adding a Flag attribute with step name using engine callback in the client sequence). My solution is quick and dirty (that is why I used "!=" in the filtering expression), but I just wanted to prove the concept and show the possible direction.

The downside of this is that you need separate Report plugins (but you could define one and duplicate it in the plugin callback; or, in general, alter plugin configuration using plugin callback). I used the custom plugin configuration "Multireport" to not mess with the default configuration. Also, you need to set somehow Attribute in the Result property (if you want to follow what I did). I used client sequence engine callback.

Perhaps it can be optimized, but the main limitation is that you can use only the Result property for filtering.

[Oli_Wachno]

Hi Michal,

nice approach. Since I am using cusom result processing anyway, I might be able to leverage this approach!

Cheers

Oli

[Oli_Wachno]

For the record, here's the approach I am trying to implement:

• for each section of Test Steps which have to be executed together, a seperate sequence is created
• a main sequence calls each of these subsequences in a new execution using a dedicated custom entry point of the model. I'll have a custom step type based on a sequence call type.

Challenge here is that the execution of the seperate section will create a standard report, which I could avoid by just deleting the ModelPlugin Callbacks from the custom entry points which is something I don't want, because I need them for different settings.

I think I can solve this due to the fact, that I am only using custom reporting anyway, which only uses the results produced by specific TestTypes.

This might work out and scale nicely

I'll give an update how it goes. Thanks for your inputs so far!

[python007]

---

@Oli_Wachno wrote:

For the record, here's the approach I am trying to implement:

• for each section of Test Steps which have to be executed together, a seperate sequence is created
• a main sequence calls each of these subsequences in a new execution using a dedicated custom entry point of the model. I'll have a custom step type based on a sequence call type.

Challenge here is that the execution of the seperate section will create a standard report, which I could avoid by just deleting the ModelPlugin Callbacks from the custom entry points which is something I don't want, because I need them for different settings.

I think I can solve this due to the fact, that I am only using custom reporting anyway, which only uses the results produced by specific TestTypes.

This might work out and scale nicely

I'll give an update how it goes. Thanks for your inputs so far!

Hi @Oli_Wachno, I'm facing the same Problem. Do you have any news. Could you solve your Problem with your described solution?

---

[Oli_Wachno]

Nope, no final solution on this.

Luckily the actual use case has not materialized yet, but it this feature is not rejected yet.

Since this discussions, expecially our PlugIn structure has grown a lot, let me re-check my notes, I'll answer asap

[Oli_Wachno]

Constraints:

- no modifications to the standard ProcessModels

- proprietary framework functionality to be added using ProcessModel PlugIns

- usage of a proprietary Result Generator

Ramification:

"have a dedicated (Main) Sequence started as a new execution using a ProcessModel for per execution TestReport" doesn't work due to the way ProcessModel Plugins are configured --> I don't think I can have a different configurations used for different executions 

This would be neccessary in my case, since some of the PlugIn must not be executed by these sequences.

Possible Solution: (a bit similar to what Michal suggested )

Have an extra custom step type at the beginning (and maybe the end) of each section of steps that has to go to a seperate report and have the Result Generator handle this information.

Implications:

loose coupling of actual TestSequence and ResultProcessing. Yet luckily only uni-directional 🙂

Hope this helps at least a bit...

BTW: Thanks for asking this question, it made me reconsider this issue