what can happen when sbrio runs out of memory?

[Pojoaque40]

My company has several sbRIOs in the field.  We are data logging various parameters to the onboard drive (flash?).  Due to a bit of incorrect code on my part the old files were not being deleted and the rio ran out of memory.  When this occured the rio locked up at it's present setting, leaving pumps and solenoids open and generally making a mess of our system.  Rebooting turned off the pump and solenoids, but the rio was unable to read files from the drive, lost it's web server, lost the fpga, and was generally unresponsive to any attempts to control it.  Erasing the files made the system functional again.

I've since corrected the code, but could someone explain to me why running out of storage caused so many problems, and are there any other symptoms we could expect to see if this happens again?  Thanks. 

[Smootastic]

Hi Pojoaque40,

As long as you've implemented checks on the remaining size for your storage, you should be ok.  You can use the Get Volume Info VI to get the number of free bytes remaining.

I can't think of any expected symptoms for out of memory issues, but ensuring it's prevention is the best way to go. 

[Spex]

Pojoaque40,

What is important to remember with a RIO based system, is even if the RT portion of the system goes into a failure mode because of a memory leak or unexpected memory issue, the FPGA portion of the app will continue to operate.  This could mean that output states are locked into their latest value, or a PID controller implemented on the FPGA continues to operate at the latest set point, or some user defined shut down on the FPGA could execute.  This "FPGA shutdown" could be enabled by implementing an RT heartbeat to the FPGA, and if the FPGA notices that heartbeat stops (because the RT app is rebooted/crashed) then it could go into a failsafe mode, stopping control loops, setting output values to safe states, and setting an indicator output that calls for help in some way.

It sounds like in your case, the RT app stopped running because the File I/O VIs were returning errors (or maybe it hung completely), but the FPGA VI continued to output values, causing mayhem.

Another option (the best option is to avoid running out of memory... ;), sounds like you are already on top of that), LabVIEW Real-time includes a feature called Watchdogs.  The watchdog has the ability to reboot the controller should something go wrong and the watchdog times out before it is reset.  If the controller runs out of memory and hangs the watchdog will recognize the hang and reboot the controller (in LabVIEW RT 2009 and later)   During this timeout period and the reboot (depending on your FPGA reset settings) the FPGA VI could still be running, and unless you program it to go into a failsafe mode, that application could still be causing unwanted havoc until the RT VI is back online.

 Cheers,