Teststand Application either crashes or hungs

[Nathan Vaidyanathan]

We use Teststand to run Overnight Tests. We have successfully running for almost an year. Currently i am experiencing issues related with crashing. Either Teststand crashes altogether or hungs in one sequence. The issue is so random, i cannot fault any particular sequence or DLL's. The same sequences and DLL's was running without any issues for a very long time. I would like to identify what is causing Teststand to crash. When i look at the Event Viewer in my Windows XP control panel, i can see the following messages at different times

Application Hang: Hanging application SeqEdit.exe, version 4.2.0.134, hang module hungapp, version 0.0.0.0, hang address 0x00000000.

.NET Runtime 2.0 Error:Faulting application seqedit.exe, version 4.2.0.134, stamp 49e4fdc7, faulting module unknown, version 0.0.0.0, stamp 00000000, debug? 0, fault address 0x20736920.

.NET Runtime 2.0 Error:Faulting application seqedit.exe, version 4.2.0.134, stamp 49e4fdc7, faulting module unknown, version 0.0.0.0, stamp 00000000, debug? 0, fault address 0x29a89472.

I have a 3GB RAM, with lots of free hard drive space. This is not a memory leak issue. I have traced the memory usage using performance monitor and i do not find any issues. Is there any other tools out there to identify which particular module is causing this issues?

[asbo]

Did you upgrade your operating system, TestStand, or really any other software recently?

[Nathan Vaidyanathan]

No i have not upgraded any software in the system. It is exactly the same. Is there any tool we can use to montior Teststand performance. I need a log of what happens during my test run. I need to know which DLL is causing the issue. From there on, i can look at fixing the DLL, if it is one of mine.

[dug9000]

Did you upgrade or change any hardware? Can you reproduce the problem on multiple machines?

You might try running things in a debugger like Visual Studio or Windbg. Another option is to try narrowing down what parts of the code are leading to the crash by cutting out as much code as possible from what you are running where you can still see the crash happening. If you can narrow it down to a small enough section of the code, it might be possible to determine the problem by inspecting that small section of the code carefully.

Hope this helps,

-Doug

[asbo]

Using NI-Spy might be useful as well.

[Nathan Vaidyanathan]

I still have this issue for almost three months. Every week, i am seeing a minimum of 2 or 3 crashes. The crash is not occuring at the same place.  I tried isolating sequences, but no success so far. The common symptoms are listed below:

1. Teststand is stuck in a step and we cannot terminate the sequence nor abort it. 
2. Teststand is executing normally with an Error dialog "Sequence Editor has encountered an Issue,Teststand needs to close". If this happens we have managed to complete the entire test run without clicking "OK" button in the dialog box.
3. Teststand crashed and there is no sequence editor window open except a dialog box saying Sequence Editor has encountered an Issue"
4. Testand hanged while generating the report
5. Testand gave a system level exception for a step and then it cannot terminate nor abort
6. Teststand is stuck when trying to launch a sequence file in a new execution. I hope this is fixed by following recommendation in the known issues (1644425)  http://zone.ni.com/devzone/cda/tut/p/id/8681#164425_by_Category

I do not get the same symptom in a row. Even with all these issues, i was able to get successful test runs for 2 or 3 days a week, so it is not like i am running some faulty sequences or DLL's. I tried running windbg tool, but i still could not make out why it is crashing, since i am not an expert in analysing those logs. If you have seen this kind of crashes or hangs please share your experience here. (I am not sure whether i am pushing Teststand too much since we are running sequences for almost 15 hours continously)

[RayFarmer]

Hi,

What are your code modules written in?

Also you didn't say what TestStand version you are using?

[Norbert_B]

Ray,

reading the first post gives me the impression that he is using TS 4.2.0.

But the question of the modules is a good hint. So in addition to tell us what kind of modules you use, you can tell us:

- What are the loading/unloading options? Do they have effect on the kind/number of issues you get?

- Since TS also hangs during a step: Those steps do have a module included? Can you verify if the module is running in an endless loop or plainly hangs (this would explain issue #1)?

- Do you encounter similar issues with default examples installed with TS?

- Do you encounter those issues in the NI Sequence Editor and/or in your own custom UI?

- What is CPU load/memory allocation during the hang? I have the impression that the system itself is still responsive, therefore you should be able to check this....

- Can you create a Performance Monitor log? Configure it to monitor CPU load, memory allocation (private bytes), handles and page faults. Is there any obvious "peak" just when the crash occurs?

Since the system was used before for a long period without the issue:

- Can you really eliminate the possibility that there has been an update of a component like the OS? So does the system have network access? Is it possible that there has been a security update patch for the OS?

- Is it possible that a system component got damage due to heat or simply "by time"? I find it possible that such random crashes are connected to a defective RAM module....so do other applications, if executed for long time which do allocate/deallocate memory more often, crash as well?

- What happens if you use another PC with the same software setup? Does it crash/hang as well?

I hope that those questions to help you in focusing and finding the root cause of this issues.

Norbert

[RayFarmer]

Norbet,

Silly me, I had looking at the date difference and not really taken in that this infact is the same Post and not one that has been hijacked.

[dug9000]

Do you have multiple threads running either within TestStand itself or within your code modules? It sounds like you might have a race-condition and/or memory corruption. If you have multiple threads running in teststand are you accessing any shared data with them such as global variables either within teststand or within your code modules?

Since this problem didn't used to happen, something has to have changed on the machine, either via a Windows Update or some other mechanism.

There could still have been a race condition all along that has only become more apparent recently due to subtle changes that could affect timing of the various threads.

Have you tried to reproduce the crash while debugging the process (ideally with mixed mode debugging)? If so what does the call stack look like?

-Doug