watchdog timer

[eward]

I have a multithreaded application that I would like to add a true watchdog timer to. I basically want a digital signal coming out of the daq (or another device) that will act as the program heartbeat tied to our interlock system.  I want this signal to tell the outside world that everything is working with the program and if any failure(caught in infinite loop in one of the threads, application crash, application hangs,etc.) happens this toggling signal goes away.  For some of the threads there are inherent stops in the code(i.e..  a pop up box asking for user inputs or delay() calls, etc.) 

A partial solution is instantiating the DAQmxCreateCOPulseChanTime() function creating a infinite pulse that runs at startup and stops during the exit routine.  This pulsing would stop if the application crashes (I've tested it) but not if the application somehow hangs or gets caught in an infinite loop.

It seems that I will need a separate heartbeat for each thread that I will manually have to toggle but that may turn out to be difficult because of the stops in code. 

Does Labwindows offer anything that will work for this solution or does anyone have any experience implementing this kind of thing? 

thanks

[menchar]

I have done exactly this for a satellite manufacturing/launch support power system.  The system sourced a 24VDC signal with an NI I/O card that was daisy-chained to all the other equipment connected to the satellite and if the power system dropped this DC signaling line, everything else powered down/disconnected itself from the satellite.  The power system would then shut itself down.

You can use an async timer, which runs on a separate thread from any of the threads in your process (well, that kindof begs the question - the async timer thread is started/managed by the CVI runtime but the aync timer thread must then be part of your process).   The async timer callback should get invoked at timer expiration regardless of what your other threads are doing.  You just need to be sure the timer gets reset somehow, regardles of what you process is doing.

When I implemented this back in '99 I discovered an error in the NI DAQ library that caused a deadlock when multi-threading, but NI has long since fixed that up.   It only occurred when I was trying to drop the DC signalling line from the timer callback at the same time as one of the other threads was in the DAQ library to do a data acquisition.  The scheme was solid - it was used on the successful IntelSat 9 launch from the SeaLaunch (RIP)  platform.

We also implemented a heartbeat scheme for a computing cluster on a recent project, this used messages over pipes rather than an analog signal.  We used an async timer for this too.

It's a mistake to crank down heartbeat timing too aggressively -  we found that we needed to find a timer value that was exceeded only in the case of a true failure as opposed to normal variation in process times.  But once you've decided you do have a failure, you want everything to shut down ASAP.

Menchar

[eward]

Thanks for the response Menchar,

But isn't creating a async timer for the heartbeat inherently wrong?  This is working on it's own thread unconcerned with what else is happening.  Let's say one of the other threads hangs talking to another device and hoses the control, yet this heartbeat would keep going in this case.  I want to be able to achieve a watchdog timer that would running in all these kinds of scenarios. 

[menchar]

Well, a watchdog timer is simply a timer that needs to be periodically reset to maintain "normal" operation.  So you do want a timer that will fire reliably when it's expired, regardless of what else may be going on (or not)  in your system.  An async timer is more likely to reliably fire than a normal timer, since it will fire when the async timer thread is active which is mainly dependent on thread scheduling by the OS, not on when you have your app process events.

The issue here is that you need to have your "normal" process periodically reset the watchdog timer, to prevent it from firing and the timer callback then closing down your process/system.

That may not be so simple, in that if you have multiple threads in your process, any of which could potentially hang up, how do you watchdog all of those threads?  You could do a separate watchdog timer for each thread.   If any of the watchdog timers fires, you shut down.  Or, you could have each thread set a signal of some kind that some other, dedicated thread montors and it then resets the async timer if all of the threads periodically indicate they are alive, multiplexing the timer, so to speak.  You could use Win32 events for this, or semaphores, or interlocked variables for example.

Then again, maybe I don't understand your design. 

Menchar

[eward]

No I think you understand what i am looking for, and what you described (all threads refreshing the async timer thread telling it that it is still alive and functioning) is what I eluding to in my first post.  It just proves difficult and i may not be able to achieve what i am looking for due to things in each thread such as delay() calls and window prompts waiting for user input.  These put natural stops into each threads' availability to do something at a particular interval at all times.  Maybe I can't accomplish exactly what I want to do, but I just wanted to see if there were any creative solutions that anybody had thought of for a problem such as this.

thanks for you response