Wait(ms) becomes highly erratic...!?

[Scott_Little]

I've got a complex data acquisition VI that runs for days on end. Recently, it had been running for 12+ days and had accumulated more than 70,000 data points when something impossible started happening. One of the things this VI does is sequence some valve operations to drain, fill, and drain again a small vessel. I have programmed these actions using a Flat Sequence structure and in each pane along with the necessary command to actuate the valve, I have also put a Wait(ms) function with a constant wired to it so that the vessel will drain/fill properly. These constants range from 5000 ms to 20000 ms and the timing is not critical. This thing has operated flawlessly for months and months and months in the past.

This time, however, one of the 5000 ms Wait(ms) functions starting varying wildly. I timed a few of them and got consecutive values like 1.85 seconds, 11.3 seconds, 5.2 seconds, etc. Thinking maybe that I'd finally suffered from some memory leak problem, I rebooted everything and started a new, fresh run. Exact same problem right out of the chute!

Any ideas out there amongst you learned Labview experts?

Thanks

[tbob]

Did something change in your OS? Did you add a new program or new hardware or drivers? Sounds like the CPU is interrupting labview, making the wait time vary. You could have a virus. Use Task Manager to see what is hogging the CPU time.

[parker]

Keep in mind that LV is data flow based and capable of running concurrent operations. When placing the simple timer in a pane that includes your operation you are not assuring that the timer is going to start at the same time as your operation. I've avoided this in the past by either placing the timer in a previous or subsequent pane or by placing a subsequence in the sequence and passing a error line through the sequence.

[Scott_Little]

tbob,

Yours was a good suggestion....but Task Manager tells me that the only thing running is Labview and the CPU Idle Process. Thanks

[Scott_Little]

parker,

This may well be at the heart of the problem. In this VI we have two independent While loops executing away asynchronously. One has a nominal 15 second repeat interval (controlled by a Wait(ms) function and the second one operates once every 30 minutes and contains the sequence structure where my problem occurred.

The hitch is this: The 15 second loop has a numerous-channel analog input scan function that "paralyzes" Labview (no other concurrent processes can execute) for about 3.5 seconds once every 15 seconds. Since there are an integer number of 15 seconds in 30 minutes, it occurs to me (now) that this hodge-podge may work well some of the time but then after a long time of running, the phase of the 15 second loop will gradually drift w.r.t the 30 minute loop until this 3.5 second dead period is falling right on top of the 5000 mS delay that suddenly became erratic.

But if that was the case, then it seems like I'd see delays of 5.000 seconds +/- 3.5 seconds....and I definitely saw some delays longer than even 10 seconds when I was observing the problem recently!?

Sigh.....

Thanks for the input.

[m3nth]

Well... it all _sounds_ simple enough... is it simple enough to post a screenshot of?

[Scott_Little]

OK,

Attached is a screen shot (.gif format) of three of the frames of the long Flat Sequence where my problem occurs. The first frame writes a 1 out to bit 0 of the digital I/O which opens the dump valve...and a Wait(ms) function waits 20 seconds (that seems to work OK!).

The 2nd frame closes the dump valve and waits 5 seconds for the balance to settle. That's the delay that worked fine for months on end then, last Tuesday, became so erratic that I observed times varying from 1.8 to 11+ seconds for that delay.

The third frame talks to the balance via RS-232 to learn the tare weight....etc.

As I mentioned earlier, this Flat Sequence is contained in a While Loop whose execution is controlled by a Wait(ms) set to 30 minutes (i.e. 1,800,000 mSec). A second While Loop that executes every 15 seconds (also timed with a Wait(ms) function) has the big data acquisition thing in it that causes Labview to freezeup (i.e. 100% CPU occupation) for about 3.5 seconds each time.

[johnsold]

Scott,

Perhaps a different timing approach would minimize your errors and eliminate your "phasing" issue. Your application seems to be a good candidate for a state machine architecture (Case structure inside loop). Look at examples and search the forums for "state machine" for more information.

For the timing read the Tick count. Add to it the number of milliseconds to wait to create an "End of Delay" time value. Periodically (in a loop) read the tick count again and compare to "End of Delay." If the current time is greater than or equal to the End time, change the state of your valve, or whatever. This way you can have multiple delays running simultaneously if appropriate and the worst case "phase shift" is the time it takes to send your digital out and get to the next state.

One caution: The tick count is a U32. It will overflow after the coputer has been running for about 49 days. If your system can run that long without rebooting the computer, the overflow case needs to be addressed.

Lynn

[m3nth]

My initial thoughts on seeing your screenshot...

1) I believe the way your delays are set up should work fine for your application, if their sole purpose is to delay moving to the next frame by more or less a fixed amount, and that fixed amount is much larger than the execution time of the other code in the frame.

2) There is obviously something else affecting the delay time (it is most likely not a function of the wait timer itself since your other delays work fine). One guess is the other complex acquisition functions you are running in parallel. Something to look at would be not only to look at the task manager for viruses which you did, but to watch it during the times when you're having the screwup. See if the CPU gets pegged for an amount of time that can be correlated with your delays.

3) Even though I stated in item (1) that your delays should work fine, they do not help you troubleshoot the problem very well because they happen concurrently with other code that is being executed. If you want to nail the timing error do this:

- Separate the timers in your flat sequence into their own cases.

- Wire the output of the Wait function out the bottom of the case it is in and build a cluster or array of all the millisecond timer values once the flat sequence has had a full run.

- You may also add a case after the code you're executing and before your Wait function, with a Tick Count function, and wire that out the bottom of the flat sequence.

The purpose of this is to give you exact timestamps (millisecond timer values) for each event. The timestamp right before the code starts to execute, the timestamp right after (before the delay), and the timestamp after the delay, which should be synonymous with the timestamp right before the next piece of code executes.

When your code is separated out without any parallelism, what you'll probably find is that the wait functions are doing just fine, that the timestamp in milliseconds is exactly 5000 (or whatever constant) + the initial timestamp that was generated before the Wait started. Also what you'll probably find is that your code in between the Wait functions is going into lala land based on yet to be determined software/hardware interaction, having to wait on the cpu or the acquisition hardware or something like that. For example, you might see that the code in between the wait functions normally takes 3ms to execute, but when your process is screwing up, that the same code is taking 2000ms to execute instead of 3ms. You will be able to determine this amount precisely by examining the difference of the timestamps (millisecond timer value) before and after the code is executed for both valid runs and for runs that have screwed up. If there is a large period of time between screwups you could possibly log the timestamps as part of your dataset so they may be examined later.

Something else worth mentioning is the Wait Until Next ms Multiple function. If you use this function for both your hardware DAQ loop and this other screenshot you posted, it may be possible to attain some sort of minor synchronization which could make your problem disappear if it's only a minor synchronization error with your other loop. That might be a long shot but it's something to consider once you understand what the Wait Until Next Multiple does.

[m3nth]

---

@johnsold wrote:
Scott,

Perhaps a different timing approach would minimize your errors and eliminate your "phasing" issue. Your application seems to be a good candidate for a state machine architecture (Case structure inside loop). Look at examples and search the forums for "state machine" for more information.

For the timing read the Tick count. Add to it the number of milliseconds to wait to create an "End of Delay" time value. Periodically (in a loop) read the tick count again and compare to "End of Delay." If the current time is greater than or equal to the End time, change the state of your valve, or whatever. This way you can have multiple delays running simultaneously if appropriate and the worst case "phase shift" is the time it takes to send your digital out and get to the next state.

One caution: The tick count is a U32. It will overflow after the coputer has been running for about 49 days. If your system can run that long without rebooting the computer, the overflow case needs to be addressed.

Lynn

---

This is a good idea but I would also add a word of caution that it would be possible to implement this without fully understanding the problem and wind up with unintended/undesired results. What this approach would be solving is the fact that the hardware calls which seem to take a fixed amount of time, can actually vary a lot. The problem I could forsee with this (which may or may not apply) is that you tell the code to go off and do something--for example the code in the second sequence of the flat sequence structure. If you have the timer value that was generated right before that was called, you would be able to have a fixed delay from the START of that command, whether the actual execution of the command takes 3ms or 2000ms. The problem is, you're only going to know that you told the command to execute at a certain time, which doesn't mean it actually did if there was a processing delay or something like that. So on the calls when it starts at the time you tell it to, this approach will give you a fixed delay no matter how long it takes, UNLESS it takes longer than your delay is set for--in which case it will take as long as the function takes to execute and your delay won't execute because it's timer value has expired. On the other hand, for calls where you tell it to start and the delay occurs internally to the cpu or hardware BEFORE the call actually starts, then you might wind up with a smaller delay than you want. For example, if you want the command to execute at time 0 and have a delay for 5000ms after it executes, and the you tell it to start at time 0, your delay will still end at 5000ms, but the command might not have executed for the first 2000ms. Which means in reality, you will only have had a 3000ms delay following the actual execution of the command. Which is all to say... you can use the approach mentioned above, but you still may run into timing problems if you haven't gone to the effort to fully understand what's going on.