CRIO fails to run at power-up, but OK if reset

[JF-NRAO]

Hi all.  We have an application that we have built and deployed for standalone operation on a CRIO 9022.

It works fine if we reset the CPU with the reset switch.  It comes up and runs properly.

If we power off the CPU and then power it on, it does not run (at least the FPGA does not drive the I/O)
  If we then reset the CPU with the reset switch, without doing anything else, it comes up and runs fine.  Any idea how to fix this?

The NO APP switch is correctly set off.

We've tried changing from using the mode where the bit file is part of the rt executable, to the mode where we flash the bitfile to the crio, with no luck.  In another post long ago someone mentioned that they increased the time delay during bootup to allow some third-party hardware to become ready before initializing everything.

How do you do that?  Is there something we can tweak in the init files somewhere, or is this done with timers in the VIs thenselves?

I posted this in the labview board by mistake, but I thought it better here.

John

[Chris.N]

JF-NRAO,

There is nothing in the ini file to add a delay to the boot up process, so if you wanted a delay it would have to be in the VI itself.  As for troubleshooting what you are seeing, it would be beneficial to do two things.  Firstly, it would be nice to have a while loop in your RT code that toggles the User 1 LED so we can see when the RT exe is running.  This can be done using RT LEDs.vi with LED number set to 0 and State set to 3 inside a while loop with an appropriate wait (about 500ms or so).  This way, we will be able to visually see when the RT exe starts.  Secondly, it would be beneficial to make the exe debuggable, as outlined here.  We can then use execution highlighting to see what the RT exe is doing when you remove and add power instead of using the reset button.  Hopefully, between these two methods, we will determine if the RT exe is running after the reset button and after powering on and off.  Also note that it takes some time for the RT exe to start so if you don't see the User LED blinking right away, give it some time.

Cheers,

[JF-NRAO]

Thanks, Chris.

We'll do these tests and see what happens.  We did make a smaller test application, which as is usual in small test applications, works fine with either the reset method or the power-up method.

The good news there is that now we are sure we are building/installing it correctly.

John

[AndyClegg]

John,

I'm not an expert on this, but we also used a 9022 in a standalone operation and noted a few things to give consistent operation at start-up.

As suggested, seeing how far through start-up it gets using the user LEDs is useful, just to see if it is some sort of software bug - perhaps related to initialisation of variables that is perhaps different after a hard power cycle. We did have a software bug which seemed to result in problems on hard power cycles (it was related to the opening of file on the 9022, but which didn't close previously depending how the application was shutdown).

It sounds like you have looked at the different FPGA start-up options, but we did find there were quite a few places where start-up settings related to the bitfile could be configured. I remember this white paper being useful for that. There's also this which may be something you have checked.

It might be useful to look at your CPU loading during start-up - I remember under some circumstances where the application didn't start properly the CPU would be flat-out (>80%), far higher than the norm for the application. You should be aware that the after a re-boot, the CPU could be doing a lot during start up, and perhaps this is the idea of putting in a delay to prevent the application moving to its CPU intensive part before it is stable after a reboot. For our application the CPU would be high for the first 10secs before stabilising (we had bitfile loaded by rtexecutable - which I think requires some time).

Finally, one thing we found that made our installations (seem) more consistent, was the clearing of the cRIO build cache (Tools | Advanced | Clear Compiled Object Cache ... ).

Hope this helps,

[JF-NRAO]

Thanks!  We had to drop this to work on more pressing matters, but we will be getting back to it once we all get home from the field.  In a week or so.  We have lots of food for thought!

John