Trends stopped working

[Michelle_jsdy]

Can you archive the original trends and delete them from the databass, can then create a simple process to log data to database to see if the corresponding trends can show up in MAX and trend object?

[mjrx]

That didn't work either. The only thing I can think of now is try to uninstall then reinstall lookout. Problem with this is, it doesn't really tell me what the problem was/is. There must be a better solution. I like Lookout, but this no support policy for it will prevent me from ever using it again.

[wetback]

I'm having the same problem, Lookout locks up, have a program dump but who can I send it to.

did you have any changes to the firewall or security settings, I had that problem once that stopped the trends.

[Michelle_jsdy]

Please post it here: ftp://ftp.ni.com/incoming/

[wetback]

Michelle,

I uploaded a dump file ‘Lockup JP Water’ to incoming, could you analyze it and see what Lookout is doing at that time?

We have been experiencing lockups like this since we went to 6.6. This happens on all our servers.

One symptom is the clock stops on the computer, there is no response from Lookout, all the services that should be running, are.

We are using windows 7 64 bit. Dell Precision T5810 E5-1600 v3 E5-2600 v3 CPUs. I will get you the process files, screen shots, or anything else you might need.

[Michelle_jsdy]

Yes, I will have a look. I want to ask can this behavior be reproduced? Or just happen irregularly?  

"We have been experiencing lockups like this since we went to 6.6", what is your original version? Did you modify your process before?

What's more, what's the basic function of your process?

[wetback]

That would be greatly appreciated. I am trying to solve the issue with Lookout locking up.

It can’t be reproduced, catching a dump looks like the best way to troubleshoot it. If you would like me to get specifics for you at the time of failure, I can.

I got involved 5 years ago after my predecessor already went to 6.6, he had 16 years’ experience at this installation, so he saw Lookout evolve, we worked together for some time and couldn’t fix it. The process has been modified all along the way, no one instance caused it to fail. I had thought we might have hit some threshold that caused it, but we are all within the operating parameters. We are using Lookout to monitor and control a water purification plant.

We have three servers on line with three backups each running a different process file, and they all experience the same problem, not at the same time. All machines are Windows 7 64 bit high end workstations. We regularly change out the computers, so they are running on the latest hardware, unfortunately Lookout cannot take advantage of the multi-processor platforms.

When a server locks up, there is no ‘not responding’ message, we lose the trends on the clients, which makes me think it might be Citadel related, the mouse cursor spins, and can be moved around on the screen, panels cannot be moved within Lookout, the computer clock stops, Lookout is not responsive. In Task Manager all the Lookout processes show the correct state as compared to a working system.

A reboot will clear things up and in the trends there will be a gap from the time it locked up to the time it rebooted.

Lookout locks up from twice a week to 4-5 times a day. When it becomes intolerable, I delete the database and it seems to go back to not locking up as frequently. We archive 7 days of trends.

I can send a process file, or all three, but would like to see what the dump has in it.

Thanks

[Michelle_jsdy]

We are now looking into this issue, seems that a mutex is taken by one thread and failed to be released. I want to ask when it is locked, what function does Lookout do? Logging data to Citadel and Clients read from it? Are there any other functions?  What about the volume being logged?

[wetback]

I want to ask when it is locked, what function does Lookout do?

If you mean while it is locked up, nothing works. Lookout pulls data from other Lookout servers, some offsite through a WAN link, sometimes slow. I can see this when I open trends on the client, some take longer to load and at times I get ‘not responding’ and the client will lock up. For clarification, the Lookout clients also lock up at times and a reopen of the client will fix it, this doesn’t seem to be directly related to the server locking up.

Logging data to Citadel and Clients read from it? Yes normally, when it locks up after a reboot there is a gap in the Citadel data.

Are there any other functions?  should I send a process file?

What about the volume being logged?   Database is about 220 MB, where can I find this ?

Do you want to do this exchange by email or stay on the forum? I really appreciate you looking into this, it has been agonizing.

[Michelle_jsdy]

OK, I prefer exchanging the information through email, you can send email to lookout.support@ni.com, if it fails, don't hesitate to tell me.

Also, if your process is complex, I suggest you simplifying it in which this problem can also happen, if we can narrow it down to a smaller scope, it is easier for us to locate the problem.