Real-Time Measurement and Control

cancel
Showing results for 
Search instead for 
Did you mean: 

cRIO-9047 Firmware Lockup glitch?

Having an issue with a cRIO-9047 momentarily hanging, it just stops for up to 4 seconds, sometimes shorter.

Details:

Using the 19.5 drivers

I've done much work on code to narrow down the problem - basic code with a Que'd loop with nothing going on, 5 msec timeout on the que, calls the default case which does nothing.

Network streams established to local PC, using NI streaming VI example code.

Monitoring Loop time on the cRIO.

In this case it may go an hour before the loop time suddenly jumps to 3 or 4 seconds, and it will repeat all night long with increased max loop times. In between will return to 5 msec and run as normal.

 

Increasing the system back to it's normal config;

Default case reads DAQmx data at 2kHz and streams to PC via network stream.

In this case it will be less than an hour and loop times will exceed 4 seconds briefly.

 

Adding on 3 cDAQ-9189 chassis's using 802 TSN, DAQmx channels being read across all chassis

In this case within 20 minutes, loop times exceed 4 seconds briefly.

 - Of course this causes sync loss on the TSN and all data is now jeopardized.

 

It does return to normal for a period of time and then again gets a spike in loop time.

 

With the basic "do nothing" code, I've monitored CPU%, and ISR%, along with loop time.

Each time the loop time spike occurs, either a CPU% went large (>85%), or an ISR went large.

 

Further info:

We have 2 identical panels setup this way. One seems to run fine - even with the DAQ running, it ran for days with no sync loss, no large loop times.

Software check between the 2 panels - same exact software - line per line.

 

I have swapped out the cRIO-9047 in the troublesome panel thinking it was an easy answer - it still is doing the same thing - large loop times with nothing running.

 

Power is good, grounding is good. No error logs on the cRIO.

 

Has anyone experienced anything like this?

0 Kudos
Message 1 of 5
(2,527 Views)

Post code

0 Kudos
Message 2 of 5
(2,500 Views)

Use NI example code, Network streams, and a que'd while loop on cRIO do nothing with a 5msec timeout.

It sat there for the weekend doing nothing and the cRIO loop time hit 3.4 seconds.

No expansion chassis, no daq, literally nothing but a 5msec timeout on a que'd loop.

0 Kudos
Message 3 of 5
(2,472 Views)

Follow up:

Switched PC's from the working panel to this panel.

Imported the NI-MAX config from the working panel to this panel.

Immediately ran into issues with the NI-MAX config - strange issues, couldn't find hardware but it was there, etc..

Tried to correct, several times over - appeared to be good but LabView would then have channel issues.

 

Long story short - resetting the entire NI-MAX config in the cRIO, and each expansion chassis, and re-doing the NI-MAX config seems to have possibly worked.

All the sudden the cRIO Loop Times seem stable and are matching that of the other working panel.

 

I'll monitor for a few days to see if it stays this way.

Hopefully this post can save someone else a week or two of troubleshooting.

 

0 Kudos
Message 4 of 5
(2,451 Views)

Some additional information that might help:

 

1. Regular loops run with normal priority, and may have to wait in line if there are many processes that are trying to utilize CPU resources.

2. Try using a timed-loop to get higher priority, and see if it resolves the issue for time-critical tasks.

3. New cRIOs come with Linux, you may ssh into the controller and use the "top" command to have an overview on what the cRIO is currently busy with.

 

 

0 Kudos
Message 5 of 5
(2,414 Views)