Intermittent issue with cRIO run process

[BenCox]

I have a process that I'm running that has been having an issue lately.

It has presumably run for many years just fine, but within the past year or so it has been having the same problem and I've been tearing my hair out trying to solve it.

The process runs a tube furnace, a flowmeter for air to pass through glassware inside the furnace, and a solenoid to choose between 2 different gasses to pass through the flowmeter.

There is a recipe off to the side to determine temperatures, gasses, and hold/transition times between temperatures.

The issue I've been running into is that, for safety reasons (this process is left unattended because it often runs for 20+ hours), there is a deviation alarm that shuts the process down, and randomly about 13 hours into the process it will trigger and shut the process down.

The furnace will have been at the appropriate temperature, gasses flowing, everything is fine, and then the thermocouple will read 0 when it previously read something like 700c, and the deviation shutdown will kick in, and then the thermocouple will read the correct measurement again and correctly record the slow decline as the process cools down.

I've checked the wiring between the thermocouple and the cRIO, and it's solid. The ethernet connection is solid. The computer is a little old, as is the cRIO, but they function normally at all other times. I've looked over the vi (which has been in use since 2010 without issue) and I can't find anywhere that this could be going wrong.

I'm not sure where else to look, and I'm at my wits end with this. The material used in the process can't be reused once the process fails like this, so I've been losing batches off and on at crucial times and it's becoming an issue.

I can include images of the process, and possibly the vi (though I don't forsee that as being an issue) if it would help.

[BenCox]

The computer is a bog standard Lenovo running windows 7, LabView 2010, and the cRIO is a 9012.

It wasn't until a year or so ago that this became a problem, it has previously run without incident.

I should also mention that the process also controls other heating jackets and gas flows in another similar process, but the windows controlling those are usually downsized when not in use (which has historically been the method). That process has also run into deviation issues.

To be honest, I'm leaning towards there being an issue either with the computer running out of memory (since the process 'deviates' at about 13 hours in of taking a measurement from 6 different datapoints every 10 seconds, my thinking is that it just doesn't have the memory to do this and dumps everything, causing the issue) or the cRIO running into an issue that causes it to read 0 on the thermocouple momentarily, which triggers the deviation alarm and subsequent shutdown before it returns to reading the correct value.

[AndyClegg]

Ben,

Could be one of a million things (sorry). But start with the basics, track the memory usage on cRIO and PC from the start over time as you get towards the 10-13 hr point just to see if it if growing (could be memory leak). Also, monitor CPU load on cRIO, is it low, does it change over time or randomly. Have you got loop execution times / loop late diagnostics logged - just in case there is some timing jitter that could be causing the alarm deviation calc to be wrong. Is the PC networked, could their be excess activity, or the PC doing a virus check or some background process.

Given that the alarm is being triggered by a spurious valve (0 deviation from the previous 700degC), then is that alarm derived from a Shared Variable or directly from the IO. Shared Variables (and Networked SVs if there is comms to the PC involved) can be prone to problems. It really depends how the alarm is derived - does the deviation just happen for one loop cycle, or does it go to zero and stay there, does the raw IO show zero as well. Could the sensor or wiring be the problem ?

Its an old system (hardware and sw versions) and things do degrade (people installing extra stuff / updates that might conflict, etc)
Andy Clegg