Stable real-time application has begun failing intermittently at startup on a cRIO-9068

[dest2ko]

I have an application that has been field-released for several months now, with good feedback from it's use. I used to load it from the LV IDE environment directly from the controller, but at some point also began using RAD utility to deploy bitfiles to configure controllers. Both methods have worked well for me and I continue to use RAD.

With a new production unit off the queue, I loaded my application to the controller as I always do and it is intermittently failing at startup. In this scenario, the "failure" is that the modules are frozen in their I/O state and will not read/write new input/output. This is for a cRIO-9068 using 9208s, 9425s, 9476s and port1/rs232 serial connection for a modbus RTU slave device. Sometimes it will work, sometimes it will not. Below are some peculiar items I pulled from dmesg when ssh'ing into the controller. 

[ 6.996228] Disabling lock debugging due to kernel taint

[    0.324911] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled

[    0.326137] of_serial 80000000.serial: FAILED to find out alias id

[    0.326759] 80000000.serial: ttyS0 at MMIO 0x80000000 (irq = 62, base_baud =3676500) is a NI 16550 (with 128-byte FIFOs)

[    0.327084] of_serial 80000010.serial: FAILED to find out alias id

[    0.327603] 80000010.serial: ttyS1 at MMIO 0x80000010 (irq = 63, base_baud =3676500) is a NI 16550 (with 128-byte FIFOs)

[    0.327925] of_serial 80000020.serial: FAILED to find out alias id

[    0.328426] 80000020.serial: ttyS2 at MMIO 0x80000020 (irq = 64, base_baud =3676500) is a NI 16550 (with 128-byte FIFOs)

Does it look like there may be an issue with the serial connection? I have attached the full dmesg to this post for further inspection if anyone can spare a look.

Thanks,

R

[Mario_C]

Hi R,

Check the IP settings of your device. If the device in question is not connected to a network, the device should not be set to obtain an IP address from the DHCP server. If your controller is set to obtain its IP address from a DHCP server, the EXE will only launch when the controller is able to obtain a valid IP address.  I found these article at ni.com, have you tried these steps?

Startup Executable Does Not Launch on Real-Time (RT) Target at Controller Reset

Best,

Mario

[dest2ko]

Hi Mario, and thanks for the input.

We have always configured our RIOs as link-locals at 169.254.0.1.

I have looked through the remainder of the steps discussed on that page, including:

- No NoApp DIP on the 9068

- Disable RT/FPGA startup are not checked

- I can't seem to find the NI-RT.ini file in the filesystem. I looked in /C/ni-rt/ but no such file exists.

- I've added a 30-second wait before booting up the FPGA, still intermittent luck on I/O function.

- I've disconnected type definitions in the build

- As far as special characters for Linux RT systems go, it still intermittently works so I'm not sure if I'm going to find much there. Also running this on other machines in the field that currently run fine every startup.

I did view the error log from the controller in MAX on a failed bootup, which provided the following under LabVIEW error log section:

####
#Date: Wed, Oct 24, 2018 09:20:26 PM
#OSName: Linux
#OSVers: 3.14.40-rt37-ni-3.0.0f2
#OSBuild: 200232
#AppName: lvrt
#Version: 15.0
#AppKind: AppLib
#AppModDate:

InitExecSystem() call to GetCurrProcessNumProcessors() reports: 2 processors
InitExecSystem() call to GetNumProcessors() reports: 2 processors
InitExecSystem() will use: 2 processors
starting LV_ESys1248001a_Thr0 , capacity: 1 at [3623260828.77271223, (21:20:28.772712000 2018:10:24)]
starting LV_ESys2_Thr0 , capacity: 24 at [3623260830.96759796, (21:20:30.967598000 2018:10:24)]
starting LV_ESys2_Thr1 , capacity: 24 at [3623260830.96759796, (21:20:30.967598000 2018:10:24)]
starting LV_ESys2_Thr2 , capacity: 24 at [3623260830.96759796, (21:20:30.967598000 2018:10:24)]
starting LV_ESys2_Thr3 , capacity: 24 at [3623260830.96759796, (21:20:30.967598000 2018:10:24)]
starting LV_ESys2_Thr4 , capacity: 24 at [3623260830.96759796, (21:20:30.967598000 2018:10:24)]
starting LV_ESys2_Thr5 , capacity: 24 at [3623260830.96759796, (21:20:30.967598000 2018:10:24)]
starting LV_ESys2_Thr6 , capacity: 24 at [3623260830.96759796, (21:20:30.967598000 2018:10:24)]
starting LV_ESys2_Thr7 , capacity: 24 at [3623260830.96759796, (21:20:30.967598000 2018:10:24)]

...and then what looked like a subsection of errors under it, with this inside the Kernel log portion:

2018-10-23T19:58:57.909+00:00 NI-cRIO-9068-01d06bf0 kernel: [    1.501551] Warning: unable to open an initial console.

[Mario_C]

Hi R,

Thank you for your quick response.  My peers here at NI have already created a service request to help you.  Randy and Andres will take it from here and we will figure out what's going on. Thank you for documenting this issue thoroughly and letting us know, we appreciate that.

Best,

Mario