rt -- cRIO disconnects from Ethernet network and stops responding to pings

[wrath_of_con]

There was a similar thread under the "LabVIEW" discussion thread but I thought I would post here.  Here is the thread in case anyone is interested:  http://forums.ni.com/ni/board/message?board.id=170&thread.id=345733

Hello,

I am having a similar issue (cRIO disconnects from the network and stops responding to pings).  I am using multiple cRIOs on a local network, and opening a Remote Panel Connection to each from another computer on the network.  There are multiple "intelligent" switches in the network.

The problem is INTERMITTENT however.  Some days, no issues at all. Then the next day, multiple cRIO systems will stop responding through the ethernet port (no pings either) and require powering off/on just to be able to connect (the cRIO box seems to be operating as normal except one must turn it off and on to reconnect through the ethernet port).

I have read the above posts about opening ports... my question is... If the ports were not open already, how could the cRIO have been connected through that switch?

I am trying to consistently recreate the my problem of "locking up" the ethernet port, then fixing it.  Any ideas?

I have also looked at the following kb describing issues with CISCO intelligent system "cRIO 900x or cFP-21xx Controller Hangs During Boot-Up Sequence of an Intelligent Ethernet Switch"

http://digital.ni.com/public.nsf/allkb/F6C7D14A7BF4541A8625713F005A20D8

FYI:

I am using a cRIO 9004 and the following:

LabVIEW RT 8.0.1
 
Datasocket for RT 4.3
NI-IrDA RT 1.0.2
NI-RIO FCF 2.2.0
NI-Serial RT 3.1.0
NI-VISA 4.0
NI-VISA Server 4.0
NI-Watchdog 2.1.5
Network Variable Eng. 1.0.0
Variable Client Support 1.0.0
 
Again no "NI-RIO" driver specifically, other than NI-RIO FCF 2.2.0, but I understand that is all that is required.

Thanks in advance!

Con

[wrath_of_con]

Sorry for the double post!

[Hesloppy]

Hi wrath_of_con,

What is your VI doing when it loses connection?  Do you see the same loss of connectivity when you run a simple "dummy" vi?

Best Regards,

[wrath_of_con]

Hi Bryan,

The vi is acting like normal... we have another computer viewing a RemotePanelConnection of the cRIO interface... shows voltage graphs, temp, some status lights, some buttons, and is writing to files...  From that remote computers point of view, the RemotePanel simply loses connection... not too unusual, usually just open another RemotePanelConnection to the cRIO and it opens where the program is continuing to run.  But in our problem, you cannot reconnect a RemotePanelConnection, you cannot ping the controller... if you look at the cRIO itself, it still appears to be functioning (relay clicking, etc.), but you cannot reconnect to that ethernet port. One must power off/on the cRIO to reconnect, however any open files will have been lost.

Its difficult to recreate these lockups... one afternoon all the boxes hooked together, no problems. We just today happened to have one box show our ethernet port lockup.  So it would be difficult for me to say, if the dummy vi worked if it happened to be running during one of its "good" times.

Any ideas?

Con

[Ben]

If there is not left over CPU the TCP/IP stuff can suffer.

Check all of your loops to enusure you have at least a "0 ms" delay in them. THe "0 ms" delay (Wait ms with zero wired) will not slow down your loops but it will let the Scheduler squeeze other process work in between loop cycles.

Ben

[viScience]

I have seen this type of problem come up many many times and as Ben has stated it happens whenever you max out the CPU.  The real problem here is two fold:

1. You should be able to reconnect with the cRIO without having to reboot

2. You should be able to specifically reserve a certain % of the CPU for TCP/IP communications if so desired.

Basically,  it is just another unfortunate oversite on NI's part which is taking a potentially fantastic RT platform and making people weary about using them.

I would say that NSV's are another example of a potentially great technology that has so many misunderstandings associated with them that they are shunned.

Even NI's own systems group prefered to write their own Networked Data Exchange technology instead of working with NSV's.

[wrath_of_con]

Thanks to everyone's responses so far. 

We are using timed loops with different priorities in the loops.

I would think if it was a problem with CPU being used too much by the code, should'nt this happen consistently?  We have used these boxes many, MANY times before, and not seen these types of lock ups of the ethernet port, however, its seems that in this particular setup we are using many boxes at the same time connected to switches over a fiber network.

For example... yesterday we connected almost 30 cRIO instruments into this fiber network and they ran all night without issues like this, until this morning when 3 of the systems showed this type of ethernet port lock up.  Some days they would lock up relatively quickly, other days the same systems that had been locking up now run fine all day.

I'm still trying to consistently recreate this problem with our current software and configuration ("consistently recreate" being the key phrase) because I want to make sure the fix actually fixes the problem.

Con

[wrath_of_con]

Thanks to everyone's responses so far. 

We are using timed loops with different priorities in the loops.

I would think if it was a problem with CPU being used too much by the code, should'nt this happen consistently?  We have used these boxes many, MANY times before, and not seen these types of lock ups of the ethernet port, however, its seems that in this particular setup we are using many boxes at the same time connected to switches over a fiber network.

For example... yesterday we connected almost 30 cRIO instruments into this fiber network and they ran all night without issues like this, until this morning when 3 of the systems showed this type of ethernet port lock up.  Some days they would lock up relatively quickly, other days the same systems that had been locking up now run fine all day.

I'm still trying to consistently recreate this problem with our current software and configuration ("consistently recreate" being the key phrase) because I want to make sure the fix actually fixes the problem.

Con

[Hesloppy]

Hi Con,

Do you think this could be a network issue?  What switches are you using in the network?  Is there any way you can determine what was happinging in the vi when it locks up the port?

Best Regards,

[wrath_of_con]

We are using a Cisco switch with fiber connections to multiple RuggedCom switches with fiber uplinks and ethernet ports that then connect to each cRIO via Cat5.  Sometimes there may be an additional RuggedCom switch added to add moer cRIOs to that chain so to speak.

Are there VI's that I can use to observe ethernet port status from a cRIO?  Heck is there a network sniffer-type VI that can be used? I am all ears to find out other ways to find out what the vi is doing when this happens.  If you mean adding more indicators or status messages I guess we can do that but that would add to the overhead of the program we are trying to troubleshoot or am I thinking down the wrong path?

Thanks

Con