Not able to do a soft reboot on a remote cRIO

[plessnick]

Hello,

I have many cRIO deployed in the field for an application.  In the last two or three months, it occurs a few times that I wasn't able to remotely reboot one of them (from LabVIEW project, reboot VI or MAX).  However, it was possible to ping the device. 

Is there any way to reboot the cRIO using a telnet command or something like that when the "official" soft reboot isn't working?  It not really practical having to take your car, drive 50 km and press the reboot button.

Thanks,

Patrick

[jareds]

Hi Patrick,

What cRIO controller are you using?  When you aren't able to reboot it, is there an application running on it?  Under what conditions are you trying to reboot the controller (e.g. the application has become unresponsive, you aren't seeing correct data, etc.)?

Also, what happens when it fails to reboot?  Do you see an error message at all?

[plessnick]

Jared,

We are using controller model 9014.  The cRIOs are controlled from a RT PXI target.  The PXI reboots every cRIOs on application start-up.  The reboot problem occurs after updating the PXI application as it is the only time that the cRIOs are rebooted.

As for messages, we do not see any as everything is real-time and remote.  We have this problem for the first time this year (system is running for the third year).  The cRIOs have been updated to LabVIEW 2009 SP1 early this spring.  Can it be related? 

We also notice that the FTP server on the PXI stops working from time to time and a reboot solves the issue (so the cRIO are also rebooted at that time)

Thanks,

Patrick

[jareds]

Hi Patrick,

How do you see the failure to reboot?  Is it just that the startup application doesn't boot, or is there something else you notice?  I don't think this is related to upgrading to SP1, but I would try to make sure that you have the correct version of NI-RIO on the cRIO ad PXI, as well as your development computer.  The newest version of RIO is 3.4.

[plessnick]

Hi Jared,

There is some handshaking between the PXI and the cRIO through Shared-Variables. If the cRIO doesn't acknowledge a complete reboot procedure after a delay the PXI raises an error.  I can also see using MAX that the cRIO isn't accesible.  My feeling is that the cRIO execute the first reboot command but doesn't start-up correclty.  However, I can ping it.  It looks like a service failed to start.

I've included screenshot of the software version installed on the PXI and cRIOs.

Patrick

[jareds]

Hi Patrick,

Have you looked into using a Watchdog timer to reset the cRIO?  You can find some information on Watchdogs in this example.

Also, do you know if the cRIO is being responsive before the reset?  Is there any initialization code that occurs in the startup VI?  If so, what does this code do?  I'm wondering if the cRIO gets stuck somewhere in the startup VI.

[speleato]

Hello Patrick,

Would it be possible to connect to the console out of one or more of the controllers and log the results to file?  That way if the controller fails to reboot, we will have some more visibility in to what isn't working as expected.

See http://digital.ni.com/public.nsf/allkb/354a5124e6a667988625701b004a77cd for information on the console out switch.

Thanks,

Sebastian

[plessnick]

Sebastian,

I do not have this possibility.  The cRIo are in enclosures located outside. 

Can the console info be redirected to a file?  Is there any error logs for cRIO as for PXI (for system exception)?

Thanks,

Patrick 

[plessnick]

Jared,

The cRIO software has a initialization VI.  This VI waits 30 seconds (so all drivers can be loaded), write to a log file and then write to a shared-variable hosted on the cRIO.  I do no see anything in the cRIO log file and a timeout is expiring in the PXI as the PXI doesn't see the update to the SV.

If the cRIO get stucks in the start-up VI, I think I will still be seen it in MAX, no?  It is not until a hard reboot is done.

I undertsand from these answers that there is no alternative way to reset a cRIO other than using the Reboot VI or from MAX.  The cRIO responds to a ping command so the OS is loaded.  I was looking for a way to reboot it from a command-line interface or to start an application from the command-line and then use MAX or the Reboot VI.

Thanks,

Patrick

[speleato]

Patrick,

I'm not aware of any way to re-direct the console out.  There are a couple of error logs,

/ni-rt/system/rtlog.txt

/ni-rt/system/errlog.txt

They won't catch problems like hangs, or show us where in the reboot the problem came up, but if the failure to reboot is due to a system exception there should be something in one of those files.

Thanks,

Sebastian