Citadel Control (nicitdl5.exe) Hangs during archive

[erblock]

Lookout 6.1

Occassionally during a Citadel Control archiving operation, especially after a system reboot, but not always;

Nicitdl5.exe goes to 99% of CPU utilization and hangs there (usually within 5 minutes of pulsing the archive start).  Lookout freezes.  If you kill Lookout, Citdl5 recovers to normal CPU utilization but then if you restart Lookout without killing and restarting nicitdl5 (Citadel Service), it hangs again and Lookout never completes startup.

In my case, this problem will repeat a second and sometimes third time after a system reboot (usually for automatic updates).  I had archiving running every hour but had to stop now.  Even an occasional manual archive start will give the symptoms (today, after about 5 minutes and 7% completion, the freeze/hang occurred.

The attached screen shows the issue.  The archive operation was started just after the second high pulse from the left.

HELP!

[Ryan.S]

I'm trying to reproduce the problem. It looks like you are archiving a huge database. What's your database size?

[erblock]

Hi Ryan,

The database is written remote and it is ~3.5 GB (365 days worth of data).  The overall size does not change much as data gets overwritten regularly.  For some time, I was actually running the archive every 15 minutes with no issues, except after reboot as discussed.  The other day, when it crashed on a manual archive, it had opened touched 85 .cdib files, 19 .cdpg files.  Looks like a total of 99 files have the "last" date/time stamp.

Ed

Just to be clear...

The Lookout database is local on the problem machine.  The archive operation (Citadel control) is archiving the local database to a remote machine when the hang occurs.  It works >90% of the time, just fine.  Even the original 3.65 GB was done manually OK after a few tries.

Message Edited by erblock on 02-27-2008 10:37 PM

[Ryan.S]

Do you delete the data on local machine after it is archived? You can archive to remote every week or so and delete archived data. Then you don't have to archive 3.5G each time, unless you do need to see long-term historical data on local machine.

I will try to make a big database and archive to remote.

[erblock]

Yes, I need cabability to see back at least 3-months of historical (Hypertrend) data at times on the local server (live) database.  I probably could cut it down, but I've always run with 365 days locally.  The archive operation only writes the amount of data since the last archive operation (and deletes some too).  On a system reboot, this should be less data actually, because the source database was not updating during the reboot.

Before I stopped the every 15-minute archives (to limit crashes altogether) the Citadel controls would run for less than 5 minutes at a time.  So it wasn't necessarily caused by the sheer amount of data that was being updated.  Although it may have had to do with how many file handles had to be opened i guess???

If Hypertrends could link to databases in addition to processes, this would ease my requirement significantly.  Hypertrend is such a cool object, I can't live without it!  If it could link to databases that were not running live lookout processes, that would be AWESOME!

This is another example I think of fixing Lookout so it shuts down smoothly on system request, such as monthly windows updates or remote administrative shutdown/restart without the -f force option.

Thanks for your help Ryan!

Message Edited by erblock on 02-27-2008 11:53 PM

Message Edited by erblock on 02-27-2008 11:56 PM

[erblock]

Ryan,

Any luck reproducing this?

Ed

[LDS@MGAS]

Hi erblock!

I´d like to know if you were able to solve this, as I´m having the same problem in my server. If you could solve it, please tell me how you did.

Thanks

Luciano.