Timing problems with Labview Real-Time on a PXI Controller

[Abmape]

Hi there,

I am sort of getting started with Labview Real-Time, so I hope some experts can give me the right hint for solving the following problem:

I want to run a real-time simulation on a PXI 8187 Controller. I have a high priority loop and I have a - normal priority - second loop for sending data to a host over TCP/IP. Everything works fine but I have lots of trouble with the timing. I noticed some time that long simulations (let's say, more than 30 seconds) took longer than expected and began troubleshooting. I found out that the simulation runs correctly for most of the time (I am simulating with 1ms step time), but after some simulation steps it becomes crazy and gets completely out of any sense - and that only for a while. After a period of time everything becomes fine again, until the next crazy wave comes.

A crazy wave means that during this time a simulation step does not begin 1ms after the last step began, but after 1.724ms or 314ms or 1.701ms or 2.006ms or or or... whatever, some random numbers between 0ms and 2.1ms. Here is an example of the described behaviour (left column = sim step; right column = step time in microseconds):

1    1724
2    314
3    1701
4    251
5    1781
6    219
7    1813
8    156
9    1863
10    98
11    1915
12    59
13    1973
14    26
15    2006
16    1984
17    2000
18    2015
19    1969
20    9

I have reduced my application to nothing - two almost empty while-loops... But the problem is still there. I have attached the VIs I am using. So has somebody an explanation for this behaviour?

Maybe the first question would just be, if the way I measure the step time is correct or not.

If the time is being measured the right way, then I do not know why the high priority loop does not start every millisecond but whenever it wants. In parallel there is only a minimal while-loop running, so... which process is producing a preemption inversion? And why does this behaviour only appear more or less periodically and for a relatively short time period, while everything is fine in the remaining simulation time?

In the file attached the stop time is set to 100 ms but you may need to increase this value up to 600,000 ms or similar (about 10 Minutes) in order to get the first wave... I can't reproduce the same simulation everytime, sometimes the wave comes sooner and some times later. Some value above 100,000 ms should already bring the fatal results...

I would appreciate any comments, cheers
Abmape

-----------------------------------------
Labview 7.1
Labview Real-Time 7.1

[Stephan_A]

Hi,

before I go to review your code, just one quick hint. Make sure that the USB port on the controller is disabled in the BIOS. If it is enabled it will cause timing problems under RT which can lead to similar effects as you see.

Regards

Stephan A.

National Instruments

[Abmape]

Hello Stephan,

thank you for your hint. I have just disabled the Legacy USB Controller but I cannot notice any improvement in the behaviour. Everything seems to be exactly as it was...

Any further idea?

Cheers,
Abmape

[aasenjo]

Abmape,

I have a feeling the CPU on your controller might be throttling itself down because of a high internal temperature. The 818X series (and most modern PCs) have a heat protection mechanism that can step the core frequency down (usually to half the expected speed) if it detects it's running too hot. I believe that in the case of the 818X series, the thermal protection kicks in if the thermal diode by the processor detects a temperature higher than 90 degrees Celsius.

I took a look at your VI and it all seems ok. You have correctly written it to avoid any source of jitter in your Time Critical loop (Memory allocations, controls and indicators,  etc) So the timing variation  is likely explained by something external, and my first guess would have been the Legacy USB interrupts, as Stephan suggested. The funny thing is that it's probably not your simulation loop that's taking most of the CPU time; it's probably the empty (free-running) normal priority loop that's making it warm and cozy inside the PXI chassis.

Try running the attached VI for a while. It's a simple benchmarking VI that will show you if your CPU is throttling itself or not. The throttling should become pretty obvious as the operations per second (or ms) should drop to approximately half or a third (depending on the stepping algorithm)

If you do find that the CPU is throttling, try this:

1) Clean the filters on the back of your PXI chassis
2) Make sure the FAN setting on your PXI chassis is set to the highest setting.

National Instruments does test and spec all their PXI controllers to function at full speed within the specified temperature range, but one has to make sure the PXI chassis is in a state capable of removing the generated heat from within the chassis (clean filters and fans not forced to the low speed setting).

As to the "crazy wave" pattern, it could be considered expected behavior if the contents of your loop are taking more than 1 ms to execute, in which case the loop misses its deadline and has to wait for the next ms multiple (you are using the "wait until next multiple" call) before it executes again.  You'll notice that the numbers you get roughly correlate to waiting until the next ms multiple. In this case, assuming the heat throttling theory is correct, you can see the loop take more than one ms only when the CPU has throttled itself down.

I hope this helps,

Alejandro
 

[Abmape]

Hello Alejandro,

thanks a lot for your post yesterday, you were completely right with your intuition! I cannot understand yet why the fan did not get automatically faster when the CPU was so hot, because it actually was in "AUTO"-mode... But now the fan is really loud (switched to HIGH) and the timing is working more or less properly. It is still not that extremely exactly timing I would wish, but it does not surpass certain limits, so it is just enough for my application.

Though your benchmarking VI did not detect any throttling... Anyway, I am happy to have this sorted out.  Thank you again!

Abmape

[Adam_U]

Abmape,

I'm curious what chassis you're using. Can you also provide me a list of the boards you are using? We use the ambient temperature to speed up/down the fan. Do you have anything restricting the airflow above the chassis (or on the sides of the chassis?)

Thanks,

-Adam

[Botag]

I am seeing a very similar problem.  I am having timing problems using the PXI-8186 controller with the PXI-1052 chassis.  I disabled the usb drive, set the fan to high, and cleaned the filters.  This did help the throttling, but it did not help the timing of my loops.  I am sending out CAN messages periodically at 15 ms intervals.  All the problems occur near the beginning of my 120 second program (I'd say within the first five seconds).  Most always when there is a hiccup with the timing, the interval spacing over a span of three CAN messages will be correct.  e.g. the spacing between the first and second message is 22 ms and the spacing between the second and third message is 8 ms.  My timed loop never finishes late (it runs every 5 ms with messages being sent every third iteration.)  I have exhausted the troubleshooting methods I could think and could use suggestions to find the source of the problem. 

Thanks,
Brian

[kmcdevitt]

Hello,

I have a PXI-8187RT in a PXI-1010 combo chassis.

How does one check or set the fan speed?

I believe that the PXI-8187 has no fan itself, so I presume you can't set any fan speed through the BIOS.

Is there a switch, jumper, or other means to set the speed of the fan?

Kevin.

[Adam_U]

The PXI-1010 chassis is not recommended with the PXI-8186 or PXI-8187 controller. The 1050 (the next evolution of the 1010) has improved cooling and can use an 8187. There is limited cooling capacity in the 1010 and older families of chassis. If you look at the Datasheet for the 8180 family of controllers (linked off the product page). You'll notice a chart on page 3 describing the environmental restrictions on controller/chassis combinations. Just as an FYI, the PXI-8190 series of controllers run cooler than the PXI-8180 family (Pentium 4-M vs Pentium-M). Even so, I would not recommend them in a 1010 chassis because of the restricted environmentals.

Hope this helps,
-Adam