Citadel Crash Notification - Lookout 6.2

[erblock]

I'm having some sort of very infrequent network event (I believe that network connectivity issues on multiple remote ascii serial devices simultaneously might be the root cause.) that causes nicitdl5.exe to hang in such a way that service manager does not detect as a crash and/or the restart of citadel also hangs.  Even with subsequent crashes in service manager set for reboot, reboots never occur.  When I log into the affected server, Lookout just has a white screen and there are no error windows on the system.  Task manager will have both nicitdl5.exe and Lookout.exe running.

So when this happens, the clients continue to see live data, but traces stop and recording of data on the servers stops, so the data history is lost.

I'm looking for ideas on ways to have Lookout pick up on this citadel connectivity and/or databasing fault such that an email or other notification might be sent by clients and/or the server itself.

Thanks,

Ed

Message Edited by erblock on 02-22-2010 10:43 AM

[JPhillips]

I have experienced this issue before as well, although rarely, but it appears to occur when our network experiences, for lack of a better word, a "glich" of some kind. If NI does come up with a way to notify us when this occurs, it would be very helpful.

[Ryan.S]

Lookout or Citadel doesn't have such kind of functionality to automatically detect a fault or stopped logging.

I'm thinking if we can periodically query the database to check the data. For example, configure a signal to periodically log data to database as a "heartbeat". And then use the SQLExec object to periodically query this heartbeat data during, for example, last 1 minute. If the Citadel stops logging or crashes, the query will return no data, or even fails.

If it fails for several times, use a mailer object to send the email as notification.

If you are interested in this way, I will try to make a sample.

[erblock]

Hi Ryan,

I think an example would be great because this is a critical hearbeat need.  Lookout executable can be heartbeated to another for redundancy, but the Citadel doesn't have this function built in.

Historical data is one of Lookout's most important features and a method to ensure it's functionality is key to that feature.  Of course an automatic kill and restart of nicitdl5.exe upon verification of logging malfunction would be a welcome enhancement to detection alone.

Best,

Ed

[Ryan.S]

This is a "raw" example.

A logger logs value 1 periodically. SQLExec object query this trace. If there is no value, or the Citadel hangs or crashes, its "failure" datamember is high. Or you can check the query result. One switch is to enable/disable the heartbeat logging. Another switch is to enable/disable the SQLExec.

You need to create a DSN first and modify the DSN parameter and the computer name in SQL statement.

Now that we don't have built-in automatic way to do so, we can use the resources on hand to programmatically do it.

Let me know if it works for you.

Message Edited by Ryan.S on 02-23-2010 02:30 AM

[erblock]

Lookout 6.5 Update:

This issue (nicitdl5.exe hangs at very high CPU utilization and causes lookout instance to have "all-white, dead" screen) seems most likely to occur when a client is using the Hypertrend cursor.

You should really look into nicitdl5.exe improvements so that is crashes instead of hanging.  That way the service being configured to restart nicitdl5.exe would actually function as intended.

Notification by email or pager is one thing, but having the system automatically recover is very important for this type of software.

Please advise what NI can do about Citadel hangs/crashes and recovery going forward into the near-term future.

Also... Ryan,

Could you please post the source file for your example or recompile with lookout 6.5?    Thanks!!

Ed

[Ryan.S]

This is for 6.5