Problem:
cFP-2220 intermittently crashes (after a couple of hours or sometimes days). Sometimes it watchdog-reboots and resumes running the program, other times it gives the 4-flashing-lights error, indicating:
The software has crashed twice without rebooting or cycling power between crashes. This usually occurs when the controller runs out of memory. Review your RT VI and check the memory usage. Modify the VI as necessary to solve the memory usage issue.
Setup:
2 cFP-2220s, 1 cFP-1804, laptop running a host program exe
LabVIEW 2011 SP1, Field Point 6.0.10
Details:
We suddenly started seeing both of the 2220’s crashing after running anywhere from an hour to about a day. This code had run for weeks without any major changes when we started seeing the behavior. We went back to code that we thought had been working well and it still crashed. We replicated it on multiple different 2220’s. Slowing down loop times seemed to help – it would run for a longer time before it crashed. We also removed a TCP communication loop that shared data between the two 2220’s (which was a semi-recent addition) and that seemed to help quite a bit, but we are still seeing the occasional crash.
The 4 flashes error suggests that the controller is running out of memory. I have monitored the memory usage and found that is not the case. I’m not sure how to interpret the CUP usage since the total CPU in the Distributed Systems Manager always reads 100% – even in “no app” mode.
Questions:
Do the attached rtlog & trace.log from the 2220 give any information on why the controller is crashing?
Are there any good debugging tools that I can use to find out why it is crashing?
Thanks!
