XML Report comes up as blank when a very large sequence is run

[Sambit_Panigrahi]

Forum,

We have multiple test sequences which me mix and match to do testing on different producst in our product line. We have no issues when we are working with small sequences ( small sequences : Which generate reports upto 12-50MB ). However when the sequences become large ( ~100-200 tests at one go , we get a blank report with the following text: 

--------------

Begin Sequence:
()

No Sequence Results Found

----------------

We notice this typically for report sizes 60MB and more. Is there a limit to how much teststands result collection memory can store ? 

My predominant report options 

-- On the fly reporting disabled

-- XML - expand.xsl selected

The same set of settings do not make any difference for smaller reports, but give error for larger sequences - So I suspect its something to do with size of report being generated ! 

[Manooch_H]

samboos -

If you're getting an Out of Memory error, please see the following forum post for an explanation of why:

Out of Memory XML Report Problem

Hope this helps!

[RayFarmer]

I created a simple sequence file which had a 100 Pass/Fail steps using the None Adaptor. These 100 steps in the form of 10 SequenceCall steps each performed 10 Pass/Fail step was placed in a For loop in the MainSequence. The loop was set to run 1000 iterations. The ReportOptions were set for XML horizonal, On the Fly enabled.

This I set running. Explorer was open at the folder where the result file was being stored. Two files were generated, the actual result file and a temporay file. I also had the Taskmanager open to monitor the performance.

As the result file got larger, about 20MB, I noticed that the size of the file was first set to zero and the data was written to the file. (It seemed like the file was deleted and generated each step result time). I also noticed that as the file got larger and larger, the storing of the step results was having an effect on the performance of the Test Sequence execution.

I left this running over night and sometime later the execution crashed. (see attached image). Before closing the dialog, I checked explorer to see what the state of the result file was. The both files was empty.

I repeated the run but this time the number of iteration was set to 500, again the execution crashed but this time the result file did have some data, rather a lot of data, over 50MBs.

I tried to open the file to check the contents but unfortunately my PC didn't seem to beable to handle a file of that size, as it was taking a long time to load, so I kill the process.

I dont think changing the iterations from 100 to 500 had anything to do with the getting the results on the second. I just think the point were it crashed was slightly different allowing the result to be transferred back to the file.

It would be interesting to find out whats going on the On the fly routine.

It also seems that On the fly seems no better that normal reports generation.  It also seems pointless generating a very large file of results and that generating smaller files would be the better way to go. Using HTML or XML a top level report file could be used to link all the smaller files together.

 

Regards

Ray Farmer 

[Manooch_H]

Ray,
 

There are known limitations of XML reporting when generating reports with a large number of results. These limitations also apply

to XML On-The-Fly reporting. The reason users might run into errors similar to the one you received is because the XML report

format generally deals with copying large strings. This string gets larger as you generate more results. The problem in particular

is that this string requires a large block of contiguous memory. You may have 2 GB of RAM on your machine but if memory allocation

is fragmented in such a way that a large enough block of contiguous, unallocated memory is not available for TestStand to store the

string in, you will receive an Out of Memory error.
 

On-The-Fly reporting is very useful in situations where the integrity of the test station is uncertain or you're executing tests

for more than an hour and you want to make sure you have a record/report of the execution in case the station goes down due to a

power outage or some other unexpected reason. With On-The-Fly reporting enabled, results are written to disk immediately following

the execution of each step (that has result recording enabled) and at specific time intervals, whereas in the case that On-The-Fly

reporting is not enabled, results are stored in memory throughout the execution and are only written to disk (all at once) once

execution has completed.

Yes, the XML On-The-Fly reporting has limitations once the number or results recorded gets to be extremely large. We continue to

work to improve the capabilities of XML On-The-Fly reporting with the hope of providing better performance in a future version of

TestStand. However, XML is not the only report format that can take advantage of On-The-Fly reporting. When executing tests for

extended periods of time that generate an extremely large number of results, it's recommended to use the HTML or ASCII report

format in the manner explained in the post below.

Report Crash

Users can also take advantage of Database Logging when the number of results becomes too large for reports to be used.

Let me know if there are any questions that I left unanswered.

[RayFarmer]

Hi Manooch,

Good explanation. I tried my example with both HTML and TEXT and both ran for the full runtime without any noticeable affect on the execution performance. Granted my example doesn't do anything other an create a large result file.

Shame about the limitation with XML as I perfer this format over HTML.

Regard

Ray

[Sambit_Panigrahi]

Ray and Manooch,

Thanks for your responses and confirming the issue. I think we have established the fact that Teststand has issues when the report size goes beyond a certain size. 

We could consider Database Reporting. Some of the other groups here use Database logging.

However our processes are presently tied to XML reporting. The other option we were considering is breaking the large ( master sequence ) into smaller sequences, and run them from a script ( python , etc ). This way we could work around the TestStand Limitations. 

We can start off sequence execution using scripts, but do not know when to fire off the next sequence. For example if we split sequence into 3 parts ( Seq1, Seq2, Seq 3) , we start Seq1 - we have no clue when Seq1 gets over and when we need to fire Seq2. We could make a roundabout way of getting this done by writing into external text files to shut-down TS and fire next sequence. 

Is there any straightforward way of doing this. Can Teststand indicate to us in any-way that it has finished executing the sequence we fired off from a command line ? 

I appereciate your inputs. 

Thanks for the help !

Regards,

Sambit

[Manooch_H]

Hi Sambit,

If you are content with taking the approach of breaking your sequence into smaller sequences and creating an individual report for each of these smaller sequences, you may want to consider using TestStand to perform this functionality as opposed to a scripting language.

In TestStand, you have the functionality of calling sequences in a new execution. When calling sequences in a new execution, you can specify whether they execute using a process model execution entry point which generates a report. You could have a top-level sequence (master sequence) which is set not to record results. This top-level sequence would then use the SequenceCall step to call your "smaller sequences" in a new execution using the Sequential Model, Single Pass execution entry point. If you want the sequences to execute sequentially (which is my understanding from your previous post) after each SequenceCall step you could insert a Wait step and configure it to Wait on Execution, then select the previous SequenceCall step in the drop-down box as shown in the image below. This will force TestStand to wait at the Wait step until the previous sequence finishes executing then move on to the next SequenceCall step.

Does this sound like a viable solution for you? Hope this helps!

[RayFarmer]

Hi,

here is a typical example (TestStand 4.2)

Regards

Ray Farmer