Corrupt Database?

[jrcamenzind]

Hello,

We use Lookout 5.1 and plot elapsed time and accumulators on Hypertrends.  Under some unknown circumstance (maybe it was a print command?), part of the program malfuctions and the Hypertrends look all messed up.  Further investigation finds that Max is not responsive and appears hung.  A reboot is required to make Max function again, then the Hypertrends clear up but show an incorrect offset.   Review of a Spreadsheet Object that records the time, elpased time and accumulators at 30 minute intervals shows a big shift in all values at the time the malfuction occurs.  Further review shows that all of the elapsed time and accumulators change at the time of the malfuction and take on values of a previous time.  From that time on, the elasped time and accumulators march on seemingly okay, but now with a big offset error.

Anyone heard of this?  Any ideas on how to fix or avoid this problem.  This really ruins the data that I am collecting so any help would be greatly appreciated.

Regards,

Joe C.

[Ryan.S]

Could you provide some screenshots to show that "Hypertrends look all messed up", and "incorrect offset" after the reboot.

How about the cpu usage when the hang happened?

[jrcamenzind]

Ryan,

Thank you for the quick response.  The link below points to a web page with 4 pictures.  The first picture shows 10 hypertrends representing all of our wastewater pump stations for a 24 hour period.  Notice that at 10:00 am the data jumps upwards.  I believe that is when the problem began.  I observed the hypertrends messed up at 19:00 .  The trends didn't look like they do in the pictures, at that point some had numerous vertical lines,like grass, some were blank, some were oddly scaled.  I then looked at the spreadsheet that records every 30 minutes and obsereved that data was still being recorded every 30 minutes.  I tried to exit from Lookout, but couldn't because it said something like "2 documents currently printing".  I migrated to the printer properties and found that no printing was actually in progress.  I did a force kill of Lookout thru task manager and then restarted Lookout.  The hypertrends still looked messed up.  I started Max application but could not get a trace of the same sources to start.  Max was unresponsive.  I rebooted the computer and I think I had to force a kill on Max.  Lookout started and the hypertrends were back but now the data had shifted.  You can see the hypertrends running again at about 19:00.  They ran okay but were way to high, so at about 21:00 I forced a reset of all trends.   The trends typically reset at 08:00.  The second and third pictures show  hypertrends of Pump Station 1 with a 10 day window and a 3 day window respectively, just for a general feel of the trend (I caused the short horizontal glitch a couple of days earlier).  The high readings a few days earlier are attributed to infiltration from high rains in our area.  The fourth pictures shows the recorded spreadsheet data.  I looks to me like when the problem occurred, that the database got lost and at line item 911 pasted the data from line item 855 , and  then marched along with an offset from there on.

I didn't look at cpu usage, but Lookout seemed very responsive, panels came up fast, hypercursors worked okay, but just messed up data.

Any ideas?

http://pages.google.com/edit/jrcamenzind/raymondscada?authtoken=0b7b52ef49187d5a2523b8cd2e79d5cae6e8...

Thanks,

Joe C.

[jrcamenzind]

Ryan,

The previous link was incorrect so I hope this works: http://jrcamenzind.googlepages.com/raymondscada

Thanks,

Joe C.

[Ryan.S]

I can't figure out which column in Excel indicates which trace or which station. Could you upload your database to ftp://ftp.ni.com/incoming, together with that Excel file? You can archive part of the database to a smaller one to upload.

Don't worry that others outside NI don't have privilege to download your data.

[jrcamenzind]

Ryan,

I am attaching a link that shows this problem occurring again.  Lookout has run smoothly for  weeks now, but an automatic reboot was forced by Windows .  The result was a jump in all of the accumulator's data.  This time the jump was downward, last time the jump was upward.  Sometimes a reboot just causes a tiny gap in the hypertrend and that is what I would always expect. 

I'm not really interested in analyzing this specific data in an effort to save or restore it, I just want to know how to prevent this.  Here are some questions that have crossed my mind.

• Have you seen this behavior before and do you know how to prevent it?
• Could this be related to using version 5.1? Might an upgrade help solve this problem?
• Could this be related to using a large number of hypertrends?  I use ten hypertrends on the attached link and ten more individuals.
• What determines when or how often an accumulator's data is saved?
• Could this be related to using a large number of acumulators?
• Can you tell me if this is a hypertrend, accumulator or database issue?

Of course, if you  need me to send you  files for analysis, then I can do that.

http://jrcamenzind.googlepages.com/raymondscada81607

Thanks for taking the time to examine my problem!

Regards,  Joe C.

[jrcamenzind]

Ryan,

I just thought of something that may be causing the problem.  Over the course of time on this project I have been making minor edits but I don't always save them.  If a reboot occurs, the programs starts with the last saved version.  Maybe the accumulators are affected when this happens.  Could that be the problem?

Thanks,  Joe C.

[Ryan.S]

I have not seen this before.

How often do you save the state file?

Is it possible that the state file causes this problem? When the lookout process is reloaded, it loads the last saved state file. Then, the accumulator will start from the last saved value. So, this may cause a big jump.

By default, the state file is saved every 60 minutes.

[Mike@DTSI]

The lookout state file... yeppie!

Have had many problems with it.  I started saving ALL (read: pots, text, etc.) to excel files.  This keeps the state file from "losing" my values.

BTW, we have ours set at 10 minutes, system will be running for weeks, then restarted and they are gone.

Good luck
Mike

Message Edited by Mike@DTSI on 08-20-2007 01:36 PM