Test Time Variation

[GeekGrok]

I am seeing a larger variation in product test time after upgrading my test environment to TestStand 4.2.1 and LabVIEW 2009 SP1. I was wondering if anyone has experienced significant product test time variation when using these versions of software.

Here's a quick background to my problem:

- previously running TestStand 3.5 and LV 8.2 with very minimal product test time variation. Running WinXP single core OS image on a dual core PC. Most VIs are reentrant and Preferred execution property is set to 'same as caller'. Using TS Batch Model model sequence.

- updated to TestStand 4.2.1 and LV 2009 SP1. Changed WinXP image to multi-core. Hoping to reduce test time execution by using multi-core OS and LV version that supports multi-core. PC and all other test hardware (PXI chassis, etc.) are identical to previous system. Began noticing a significantly larger variation in product test times when running with these new versions of TS and LV. While test execution is faster, the variation in test time has increased. Most (if not all) VIs are reentrant and some VIs have the Preferred Execution property set to 'standard' to take advantage of multi-core processing. Using same Batch Model model sequence.

I suspect something "destabilized" my test time execution when I switched to a multi-core platform. Has anyone else attempted this type of update (from single core to multi-core) and experienced a larger product test time variation?

Thank you in advance for your help,

Jim

[Norbert_B]

Jim,

first of all: Is the increased jitter in "time per UUT" a problem for the system?

I don't get the point of the 'image' you are referring to. WinXP can handle multi core and will use all cores available to it. So if you create a WinXP image on a single core, install and run it on a multi core machine, it will behave differently than on the single core. The scheduler will take all availables cores to run the tasks on and will possibly switch tasks currently running from one core to another. This is called "load balancing" and there is always a chance something like this happens. Please note that swapping tasks from one core to another is creating overhead.

There have also been changes in the TS and LV versions regarding multi core support since true parallelism (which is possible on multi core machines, but not on single core) requires additional security features. Simple reason: prevention of race conditions. So for instance, LV includes now an improved algorithm for detection and resolution of race conditions for subVI calls (non reentrant). The increased security has a pay off: the time for the algorithm to work properly is increased by about 100% (from about 1us to nowadays 2us).

There are many features to control execution of the tasks on the cores, but most often, the automatic load balance leads to the best/fastest execution times.

Nevertheless, there might be some additional jitter as you observe.

So if the jitter is something you can capo with, than i would simply take it as "as is". If the jitter is unbearable, this is definetly something we should take a closer look. If so, you should post some example code.

hope this helps,

Norbert

[GeekGrok]

Norbert,

Thank you for your reply.

Yes, the increased "jitter" is per UUT and is a problem for our production system as we have seen test times vary by as much as 40%.

As for the WinXP image, let me try and clearify: The PC for my test system in dual core. However when we first built the test system (using TS 3.5 and LV 8.2) the WinXP was installed onto this PC as a disk image. This image was taken from a single core PC. Thus even though my PC was dual-core, the WinXP image that was copied oto the PC was single core. WinXP could not update this image to dual-core, most likely do to the fact that WinXP was installed as a copied image, and not from an installation DVD. When I decided to upgrade to TS 4.2.1 and LV 2009 SP1, I then had WinXP installed on the same dual-core PC. This time, WinXP ws configured as multi-core.

The load balance you refer to, does this mean that I should disable the reentrant execution setting on my VIs and let the OS handle this? Are the changes in TS and LV you are referring to applicable to the versions I am running, 4.2.1 and 2009 SP1, respectively?

Unfortunately, I cannot supply any code exampels at this time, as I am not sure if the sequence or VIs are causingthe problem. Once I can narrow it down to a specific module (or modules), I then might be able to supply an example.

Thank you again for your help,

Jim

[Norbert_B]

Jim,

you might take usage of some tools to track down possible resource issues. You can use the Windows "perfmon" (that's the Performance Monitor as you open it by typing into "Run") to observe the overall performance of the process. You should get some instructions on this tool via MSDN.

You might want check memory usage, CPU usage (not only load balancing) and paging (page faults).

Can you still compare both systems? So can you compare results from the tools from both PCs?

What about reporting? Is it possible that generating and saving of the report is the issue? Can you track down if the jitter is really part of the UUT test procedure?

Norbert

[dug9000]

You might get some performance advantage/stability by using the CPU affinity step type or station options preference to set the TestStand threads to all run on the same core, letting labview use the other cores more for itself. It might be worth trying various combinations of settings to see what works best for your use case.

Hope this helps,

-Doug

[GeekGrok]

Norbert,

I have been using perfmon and task manager to view system performance. Not seeing anything obvious in these tools. I do not think reporting is the issue as the way I track test time is basically only during the execution of the actual UUT test sequence (any pre and post activity is not accounted for in the test time). I can also see the test step execution time vary, so I believe this is where the test time variation is occurring. Also, it is not one specific test step type that is varying.

Doug,

THank you for the info on the CPU Affinity information. After reading up on this, it sounds like this could be my problem as I simply "ported" over my test sequences from TS 3.5 (single core support) to TS 4.2.1 (multi core support). I am playing around with the CPU Affinity settings to see how this might impact my test execution time.

Thank you both for your help. You've both given me some good leads. I'll continue to do some further benchmark testing to see if I can root cause this problem and report back then.

Thank you,

Jim

[AllenP]

TestStand 3.5 supported multiple cores so it wouldn't likely solve your problem.

[dug9000]

TestStand 3.5 didn't explicitly keep multi-core from working in anyway, however it sounds like your OS wasn't configured to actually use multiple cores when you were also using TestStand 3.5.

Also, newer versions of teststand have been optimized to run better on multicore machines as well as providing new features such as the CPU affinity step types to help tweak multicore performance.

Hope this helps clarify things.

-Doug