TestStand 2010 crashed during two day test

[Scott754]

Greetings Everyone,

     We are using TestStand 2010 for evaluating our products in an ESS (Electric Stress Screening) Environment.  The test and test data are accumulated over the coarse of 2-1/2 days.  If we run the test program for a 2-cycle environment test (1-cycle = 1 Hot and 2 Cold tests), the TestStand programming is OK.  When we run it for an 8-cycle test, the TestStand crashed with a dialog box (attached) after 3-2/3 cycles.  This only started when we activated the On-The_Fly reporting option.  We are using the Text format for the data reporting.  The file size if 367KB and it seems that there is plenty of memory, so I can't see it being a memory issue or maybe it is.

     One question I have is, what is this dialog box telling with regards to the failure.  Another question is, In TestStand is there a file size limit embedded inside TestStand somewhere that I'm unaware of that could be causing this.  We have run this scenerio 3 times and got the same result as far as the point that the system crashes.

     I appreciate any and all information that you may provide with regards to this issue.  Thank you in advance for taking the time to answer this post and look forward to hearing what all of you think of this.  If you need additional information please don't hesitate to ask.

Thank you,

Scott

[V_T_S]

Try discarding the result..

Configure -> Model Options -> Enable Discard Results or Disable Results When not required by Model

[Scott754]

Hi Vidula,

     Thank you for your reply.

     I'm a little confused.  What will happen the data reports that I'm collectingdata for if I discard the results?

     Just to let you know, we were not using this mode in the beginning.  We selected this mode of reporting in order to look at the data during test.  My co-worker and myself still have a lot to learn with TestStand.  It's not like we're exceeding the memory (file size 367KB, normal 8-cycle is 815KB) of the PXI-8101.

     Here is some additional information that I didn't add before.  We are testing 4 systems at the same time.  There are some Batch Synchronizations inside due to data in most cases can only be taken one unit at a time.  One cycle of testing is approximately 6.5 hours.  We've run 2-cycle test in this mode without a problem.  We discovered this issue (see attach from previous entry) when running 8-cycle test and the reported data ends after 3-2/3 cycles of testing.  I'm most interested in what the previously attached picture is telling me.  I tried a google search and came up empty.  I don't know if clicking the "VIEW" button after selecting the "Report" tab could've caused this and if so it took a while to occur.  We have two (2) TestStands running the same client file and both crashed over the weekend.  Hope this get everyone and yourself a bigger picture of what we are doing.  My co-worker called NI last week and the customer service was clueless as to what could be causing this problem.  Its kind of funny that 3 times in a row this problem happened.

     I appreciate your reply and I'm curious as to what will happen to the data being stored in the file that is created if I select the model option you have spoken about.

Regards,

Scott

[V_T_S]

If the Discard Result option is enabled then TestStand discard the results after reporting. In this way you can optimize the memory consumed by TestStand for each execution.  

[dug9000]

When you talk about memory use you are referring to the file size, but that is not necessarily the memory that matters in this case. What about the memory use of the process? Look at "Private Working Set" memory in the task manager.

If the problem is not memory usage (check private working set before concluding this though), then you likely have a race condition in your sequences. Use the previously mentioned report option if memory use is or might be a problem. It likely won't affect anything you are doing as long as you aren't directly accessing results yourselves outside of teststand's normal report generation.

-Doug

[Scott754]

Hi Vidula and Doug,

     We ran a 4-cycle test over the last day and a half and timed it so we would be in front of the Test Stand just before the time that the TestStand would crash.  We activated the "Discard Results or Disable Results When not required by Model" as suggested by Vidula.

     Another piece of information with regards to the test program is the program is in a loop reading one of the DIO ports looking at a Relay closure in the Environmental Chamber that instructs the program to continue with the next test event.  This is were it was crashing.  Unless I'm wrong there should not be any race condition to talk about since we are running a Batch Synchronization for the looping step.

     Since the completion of testing wouldn't finish until after we left last night, we waited to look at the data that was saved after the program completed.  The suggestion of clicking the "Discard Results or Disable Results When not required by Model" in the Configure->Model Option Menu seems to help with the crashing parse, but all of the test data is non-existent, which I asked about earlier.  On both TestStands the four (4) UUT's files are all 1KB in size.  The only thing that is in each file is the appropriate header information and results of PASSED.  This solution is not good for us since we must present data to our customer.  It seems that there may be a file size limit embedded in the TestStand environment that is limiting the combined file size to 1MB.  As stated eariler in this post we have run 8-cycles (each file size = 815KB for each of 4 UUT's (2.46MB)) with On-The_Fly deactivated.  I don't know, maybe we are asking the TestStand environment too much for this application that we are performing

     We noticed something very unusual with the file size as the test was running.  While looking at Windows Explorer and having the report directory selected, up til the point that the file size reaches 300KB the file sizes would alternate between the size of 300+ and 0KB.  We looked at the Physical Memory and there is NO issue (over 400MB) even while the program is running.  Shortly after the files sizes acheived 367KB it got worst.  The file sizes stopped updating (day/time stamp) in Windows Explorer on one of the TestStand before the other starting doing the same thing.

     So, my question now, Is there a limitation to the size of the Text file that the Report Generator is producing under the On-The_Fly condition.  Also does anyone have a clue as to what the attached picture is telling us, which showed up the three consecutive times that the Stand crashed prior to now.  I appreciate all suggestions but still not sure why the stand crashed in the first place.  Thank you.

Regards,

Scott

[dug9000]

1) What do you mean by "we are running a Batch Synchronization for the looping step". Please be more specific? Is all access to your drivers synchronized so that only one thread is calling into them at a time?

2) There is no 1 MB file size limit for report files. There is no limit other than available resources on the computer. Perhaps available memory is the main resource since the full report likely has to fit in memory.

3) Did you check the private working set (there are many types of memory to look at so make sure this is the one you are looking at) of memory for the process using task manager like I suggested? If so, how much memory was the process using?

4) If it is always crashing on the call into the relay closure driver that is most likely where the issue is. What programming language are you using to make the call? Have you tried calling this driver in a tight loop and seeing if you get a similar failure?

5) That crash dialog seems to indicate an unhandled exception occurred. exceptions are thrown in many different programminging languages such as C++ and C# to indicate an error has occurred. TestStand normally catches all such exceptions when calling users code modules so they normally do not crash the process, however if the exception occurs in a thread spawned by a called code module it can still crash the process.

6) Have you tried disabling report generation and seeing if the problem goes away. Disabling report generation should help determine whether or not the problem has anything at all to do with report generation. It might not. If the problem still occurs, you'll know you need to look elsewhere.

Hope this helps,

-Doug

[Scott754]

Hi Doug,

     Sorry for the delay in responding to you last reply.

Question Number 1

Yes, only one thread is calling into at a time, but I was mislead.  Please see the attached picture of the sequence that is running just before the crash.

Question 2

The typical file size for a 2-cycle test is approximately 215KB and an 8-cycle test is approximately 815KB.  These are the file sizes when running the TestStand in normal reporting mode (not On-The-Fly).  While running On-The-Fly reporting, we observed the file sizes in Windows Explorer, while running a mock version of the client sequence and watched the file sizes increase as they should while the data is being compiled.  As soon as the accumulative total reaches 1.5MB the files stop updating.  If you terminate the client sequence, the simulation showed the four file sizes as 497K, 498K, 499K and 1K.  There was only a header in the 1KB file.  Another scenario was the four files reached 367K each and stopped updating as well.  There are times that one, two or three files will show 0KB.  My co-worker has been on the phone/email with National Instruments, but they seem clueless.  They’ve given him some examples, but these are not helping the situation.  It seems that they are running a similar scenario on their PC and not a PXI-8101.  He has sent them screen captures of the report configuration and some other requested items and has not heard back from them yet.  This problem only happens on the PXI-8101 and not a Desktop or Laptop computer simulating the PXI Chassis.  We are using the ASCII Text File report generation file as to keep the files sizes down.  It's very interesting that this only happens when using the On-The-Fly reporting.

Question 3

We are using the Windows XP version of the PXI-8101.  On board RAM is 1GB.  The memory usage is fine with over 400MB of free memory available.  We watched the Task Manager while the Client sequence was executing and the TestStand and Labview memory would fluctuate.  The TestStand would be approximately 130MB and Labview would be approximately 30MB.  There is really nothing much else running on this computer since it is a standalone and not connected to the internet.

Question 4

See the attached picture to see the sequence that is running at the time of the crash.  The language is TestStand driven since it all TestStand and Labview code.  As far as running this sequence in a tight loop to see if it crashes, we don't think it has anything to do with it since running TestStand normally (not On-The-Fly) does not show this problem.

Question 5

Just curious as to what in the world this was telling me that could shed some light on the problem that it is warning me about that’s all.

Question 6

I don't feel that turning off the report generation will fix this since running TestStand normally, without On-The-Fly reporting, works just fine without losing data. 

     Well there's a mouth full.  Let me know what you think about the answers.  In the mean time we've gone back to the way we were doing the reporting and everything is working fine.  We tried something new at the request of our VP so he could see if anything was malfunctioning during these 3 day tests instead of waiting till the end and being disappointed.  As I mentioned earlier, maybe what we are doing is a little much for TestStand to handle, but still can’t figure out what is causing this phenomenon.

Regards,

Scott

[warren_scott]

From my experience, the architecture and memory management setup for on the fly reporting in TS2010 and older does not work for large sets of data (which you have).  Our solution was to avoid on-the-fly reporting (and hope for a better overall report generation architecture in future versions of TS).

We have some similar challanges.  We made some tweaks to the process model to do "progressive testing" where we would configure and launch a test on a product, and it would basically run "single pass" on an instrument configured for cold cycle and log data/generate reports, then without user interaction start "single pass" all over again configured for a hot cycle (and do reports), and then do it again for whatever configuration of additional hot/cold/warm/wet/dry cycles we wanted.   This allowed us to get away from OTF reports and keep the TS memory footprint smaller because it never gets "days" of data stored in the results tree, it only ever gets "6 hours" of data and then flushes that memory when it starts the next cycle.

[dug9000]

Are you logging results for that while loop in the screen shot? If so how many iterations of results do you typically have? If you have result collection on for a tight loop like that you can easily be creating thousands of results a second which could easily begin to use up sigficant amounts of memory. You might want to turn off result recording for the steps in the loop and see if the problem goes away.

Hope this helps,

-Doug