result filter behavior differerent in ATML vs HTML

[david_jenkinson]

Hi,

We recently updated our teststand to 2012.  I noticed the default report format was ATML, and left it like that, because it actually looks a little nicer and has a couple nice features over HTML, what we've used. 

We then noticed that we were missing results from a step, the result text of that step being a hyperlink to addition failure data.  This step is executed in a sequence called by the main test case sequence, so the call heirarchy is

Testcase sequence

     Subsequence

           Step <-result is masked in ATML, not HTML

We have our report filter set to "exclude passed, done, skipped".  The way we have forced the Step to show up in the report, is hard code it's status to "ReportMe".  Step is only executed when Testcase failed, it's status is passed down via parameter.  It has worked for a long time this way, we've always had our reporting set to HTML.  In HTML it does not seem to matter that the status of the Subsequence is "Done", Step is reported if executed.  This is not the same behavior in ATML, step is not recorded at all.  It does execute though, I single stepped through the sequence, and made sure the ReportText contains the proper hyperlink string.

So in experimenting trying to get the step to show up the same way in ATML, I hardcoded the Subsequence step whose results are normally "Done" to "ReportMe".  In this way I can get the Step results to show in ATML.  However this has the disadvantage of having a whole bunch of "Subsequence" calls with a status of "ReportMe" showing up , on the iterations where Step did not execute (test passed). 

So I'm wondering why the difference in behavior?  HTML seems to allow results in it's report when status passed the filters underneath steps that do not, ATML does not.  ATML seems to prioritize the status filter of the callers, and masks all the substeps regardles of their status, where HTML does not.

[Danceman]

Hello David,

     First off, I really enjoy the new ATML format.   I have been looking over this issue and I tried to recreate the issue on my computer and I have only seen the behavior you discuss with the ATML formatting.  When I switch to TestStand 2010 SP1 and run with either HTML or ATML I again get the same behavior as you discuss in your forum post.   No matter what it seems like I have to switch the Status result from Done » ReportMe   in order for the reports to be generated.  (Note:  I have set "exclude passed, done, skipped" constantly on my machine at the moment.)   This seems to be expected behavior as well seeing as you are trying to exclude all tests the result in "Done"  

    To help support you further can you send me some example code that will recreate this difference in reporting for HTML vs. ATML report generation options.

Thank you,

Daniel

[Danceman]

Update:  I am getting the exact same behavior in all report options for TestStand 2010 and 2012

[ars1647]

Hello Everyone!

I am new here and don't know how to post a new question so please reply me.

I am calling a sequence in STA and cannot get report of its steps. If i run indivisual steps from the sequence than there is report but when i run whole sequence there is no report of steps in it.

I am using HTML report options and calling functions from a .Net API.

[david_jenkinson]

Hi,

Thanks for the reply .  Yes I will send a small sequence when I get a chance, unfortunately other things have my time but I really want to see this work, so it will be here eventually.

Thanks

David J.

[Danceman]

Sounds good David.  Let me know as soon as possible.

As for calling a sequence STA and not getting the Report created.   Which process model are you using?   Please verify that you are not running without a process model by going to EDIT» Sequence File Properties » Advanced tab.  Also if you could near the bottom lefthand corner of Sequence Editor you will see Model:  (Your process model).   If you would double click there and verify that your process model has sequence reporting within the model.
Lastly,  you may need to verify your setting within Sequence Editor » Configure  Report Options.

[david_jenkinson]

Yes we are running a custom process model.  I just got through a few days of porting over our process model from the old architecture to the 2012 architecture, all seems to be working ok.  This is when I noticed ATML was the default and just stuck with it, later noticing the result was missing we always had.

So I just put together a very simple sequence replicating what we are doing (or so I thought) in our production sequence, and now I cannot reproduce.  It behaves as I would expect, the step shows up in both formats.  So the question now is what is different with our sequence.  Our sequences in question are rather large so I'll probably take copies of them, delete all steps/sequence calls but the ones in question and try to duplicate from there.  I'll post that here if I have any success in replicating.

Thanks

David J.

[david_jenkinson]

Hi,

I've managed to successfully replicate this by editing the existing sequenced down to bare minimum.  With the attached sequences I get different results by merely changing the report format.  It should work if you unzip them  to the same directory.  Top.seq is the top level sequence.  I've also included a couple images of the two results.  I am still stumped by this behavior...

[ars1647]

HI Danceman!

Thank u for your quick response, I have checked all the options you sugested and they are OK, furthermore their snaps are attached. If i run a single step from the sequence its report is displayed but when i run whole sequence report of its steps is not shown. Seq. is running in a new thread in Single Apartment Thread and i think this is the root problem. As i am using Teststand since last 4 years i don't think I am missing any basic report step. I'll really apreciate any help.

[david_jenkinson]

Hi Danceman,

Was wondering if the files I supplied allowed you to replicate the behavior I am seeing? 

Thanks

David Jenkinson