Re : Re: Test Stand Shut Down with no reason

[Kobi_K]

Hello ,

I'm using Test Stand 4.2 & LabWindows CVI 9.0 .

I'v created a program that checks 30 cards in Batch mode , using Test Stand as master that calls CVI functions from dll files i'v created .

In the program there are no error nor warnings But i have a strange problem for every 50 runs of the program I'm facing 1 Complete shut down of TS with no reason or warning .

TS simply shut down and not showing any message . This problem is very disturbing considering that full check for 30 cards using TS is Approximately 3.5 hours .

The shut down can occur at any part of the run with no repeated pattern , It can happen after 3 min or after 3 hours And by doing so the user need to restart the entire test from the start because TS 

didn't produced any Report .

I'm not denying that the crash can be in the dll files written in CVI but thats in low possibility .

I'm under Win Xp and using intel core2duo L7400 and 4 MB of RAM . 

Is there any option to avoid this problem ??

Thank You , Kobi . 

[RayFarmer]

Hi,

You dont really see what your test sequences are doing. Whether you have shared resourses, such as memory, hardware.

How many test step do you that are setup to record the result?

Does the crash occur on every 50th run? Do you know what the state of your memory, checkout task manager an see if you have some sort of memory leak.

Also, how is your report generation setup, are you using on-the-fly with conserve memory enabled.

regards

Ray Farmer

[Kobi_K]

1. I'm Recording result only in few Test Steps because all the other are irrelevant ( Only things like DMM Readings ) .

2. It occur randomly it can happen every 50 Tests or Every 100 Tests .

3.  on-the-fly with conserve memory enabled , I don't know what that is,  Its my first project with TS so if u can explain it will be great .

4. is there a tool like in CVI to check for memory leaks ?

Thank You very much Ray ! 

[dug9000]

Most likely you have memory corruption issues with one or more of your code modules. This can happen very easily in C/C++ programming when incorrectly accessing pointers that have already been freed, accessing an uninitialized pointer, or freeing a pointer after it's already been freed and numerous other incorrect pointer or handle accesses. Memory corruption is a tricky bug to track down because the crash often occurs at a completely different part of the code than where the bug is and it's often random like you are seeing.

One way to narrow things down, if you are using the CVI adapter, is to use the adapter option to run code modules in an external CVI (It's under adapter configuration options. You must have CVI installed on the machine). You can change DLL adapter steps to CVI steps and due this for those as well if needed. This will help narrow down whether the issue is with those code modules or not. If the external CVI process starts crashing instead of the main TestStand process, you know the issue is with those code modules.

Basically for issues like this you will likely need to narrow down precisely which code modules are necessary for the problem to occur by trying to make the problem happen in as small a sequence as quickly as possible. Once you've narrowed it down to a small subset of your code, then you can inspect that code carefully for incorrect pointer accesses or incorrect prototype definitions in your dll or cvi step module settings.

Hope this helps,

-Doug

[frjibeau]

Hi Ray;

For industrial electronic test, i use BatchModel of TestStand4.2 and LabWindows/CVI 8.5.Into TestStand sequence i call functions of CVI dll.

At the beginning i developped a TestStand software which run on a PC which do not have a multiCore processor and have 1Go RAM. The software runs great but the execution speed was not very good.

I decided to change my PC and now i use a PC with 2 dualCore Processor and 2Go RAM.Since that time the test software stop with any reason and message,and TestStand shut down.

Could you help me.

I do not undestand why this issue.

JIBEAUX Frédéric
Test Engineer

Alcatel-lucent France.

[dug9000]

Race conditions often show up more easily on multi-core machines. The fact that you didn't see the problem until you switched to a multi-core machine implies that there is a race condition in your code. Do your code modules perhaps need a lock around calling them (i.e. is the code designed correctly for multiple threads calling into the dll at the same time) or are you perhaps accessing and modifying TestStand globals from multiple threads at the same time?

Hope this helps,

-Doug

Message Edited by dug9000 on 05-27-2010 10:07 AM

[Kobi_K]

I will try to narrow it down .

But maybe there is something like the Resource Tracking in CVI for Test Stand ? which i can run a project and these his leaks  

Because i'm running DLL its a bit tricky to find the leak . 

[dug9000]

A race condition is not the same thing as a leak. Instead you should look for things like accessing global variables from your code module without protecting that access with a lock. Both TestStand global variables and CVI global variables. Global variables include variables inside a function if they are marked with the "static" keyword. TestStand global variables are safe to get or set their values from multiple threads without using a lock (TestStand does the locking for you), but if you are inserting or removing subproperties from them, then you need to get a lock before doing so and before accessing such variables.

Other things that might lead to race conditions are the use of an instrument driver that is not threadsafe from multiple threads at the same time without using a lock. There are other things as well. Think about how your sequence is running in parallel and consider what code might not function properly in such a situation.

Hope this helps,

-Doug

Message Edited by dug9000 on 06-08-2010 08:30 AM

[Kobi_K]

Thank u Doug i will check it , for now i have pasted the at the bottom the event viewer application log maybe some as i suspected the problem is in the CVI DLL but i don't know what this Error means 

Maybe You know ?

The description for Event ID ( 0 ) in Source ( CVI ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: [11:22:25.140] [FATAL] [LRemoveItem] [..\james\list.c:543] Assertion failed: 0 (itemPosition >= kEndOfList && itemPosition <= (*theList)->numItems).

The description for Event ID ( 0 ) in Source ( CVI ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: [11:18:43.140] [FATAL] [LRemoveItem] [..\james\list.c:543] Assertion failed: 0 (itemPosition >= kEndOfList && itemPosition <= (*theList)->numItems). 

[dug9000]

It sounds like the assert indicates an invalid index is being used into a list of some sort. Is there perhaps a CVI list object being shared among the threads? If so you will need to synchronize access to it (i.e. use a lock) so that you don't have situations like one thread removing or adding something to the list while another thread is trying to access it or also modify it. Really all access and modification of shared data structures needs to be protected in multi-threaded programs, whether the data is shared in TestStand or internally in a CVI DLL, the same issue applies. There are multiple ways to synchronize access to shared data, but using a lock is one of the simplest. Other examples of synchronizing access to shared data is using Notification or Queue synchronization objects.

-Doug