Real-Time Measurement and Control

cancel
Showing results for 
Search instead for 
Did you mean: 

CRIO looses network connection

(Also posted on general Labview board )
 
I am developing my second CRIO application. This is a quite complicated real-time application that has been running on a PC and I am now porting over to CRIO for better determinism, more robust platform etc. I've done the FPGA coding and this is well tested, and now fitting in the rest of the real-time control into the CRIO-9004. I plan to use shared variables to get data to an accompanying PC for display, but have just two test variables running at the moment.
The problem is that the RT program runs for a while and then appears to hang up from the network, although the network port leds continue to flash as normal.. The FPGA is still working because it is setting outputs in response to digital inputs. Neither MAX or the test vi looking at shared variables can see it. This happens with a deployed RTstartup program, or from project explorer.
The only way to recover this is to go the rig and actually press the red reject button (or cycle the power).
 
I've been watching it with RT system manager, and although the CPU usage can go up to 100% if I have too many diagrams open, this is not definitely the cause - it just hung up from 60% usage, I'm using about half the RT memory so plenty of space there.
 
Possible causes -
There's too much code, performance needs to be improved.  - Does anybody know of useful utilities to monitor cpu performance as a sort of watch dog.
 
I am publishing shared variables from the CRIO. Viewed on the remote host, the update rate appears to slow down before connection is lost.
 
The network or vi server is shutting down - Again, I'd appreciate some help on how to monitor this.
 
Obscure fault on CRIO hardware. ??
 
Looking forward to finding out if other people have had similar problems.
I hope it's something that Labview 8.20 will cure??
 
 
Lucy
0 Kudos
Message 1 of 5
(5,887 Views)
Just an update on this in case any other users are interested in the resolution.

This query was logged as a service request with NI UK Technical Support. Lucy has SSP and upgrading to LabVIEW 8.20 appears to have resolved the network connection issues she was having.

Lucy if you'd like to add anything to this feel free 🙂

Many thanks,
Sarah

Applications Engineer | National Instruments | UK & Ireland
0 Kudos
Message 2 of 5
(5,742 Views)
Any idea why upgrading to 8.2 fixed the connection problems?  Was the problem with shared variables or general connection bugs in the cRIO?
0 Kudos
Message 3 of 5
(5,735 Views)
Hi,

When I took on the SR we took a "wait and see" approx to it as Lucy did think that upgrading might make a difference. This did make a difference and I believe the reason is just down to the usual upgrade improvements.

The shared variable was a new technology introduced in LabVIEW 8.0 so that will have been improved, made more reliable, in the latest version.

I also think Lucy changed her application a bit to manage her CPU usage and shared variables better.


Sarah

Applications Engineer | National Instruments | UK & Ireland
0 Kudos
Message 4 of 5
(5,731 Views)

8.2 itself has not solved the problem as I can reproduce it quite easily by forgetting step 3 and trying to send massive streams of data. However it has helped greatly.  I don't think there is anything specific, just loading and running faster and not crashing so often has helped ease the development process. In 8.0 everything hung when I saved a vi, or changed a shared variable and the tools mentioned below never worked very well.)

Steps I believe has helped:

1. Using Real Time Module system manager to look when the cRIO  processor and memory is being overloaded - ...... I quickly discoverd that I should close as many vi's as you can while debugging, and try not to use probes unless really essential. I've got cpu usage down to under 50% in normal running and things are more stable.

2. Use the Shared Variable Manager to check the health and status of the network variables ... I identified some buffer overflows......

3. .....take very good control of the shared variables, and don't allow them to be too large . I have implemented handshaking on the variables that pass data continually between the pc host and crio.  (an extra boolean flag sv to tell the other side to read the data, and delay new data being written until it has been read. I think this was the main problem I had: I assumed that the internal handshaking would cope, but instead big (hidden) queues were obviously building up and taking all the network capability up, leading to a disconnect that it never recovered from. I've stopped getting duplicate data or missing data too 🙂

4. also being able to search the code for Shared Variables has helped - but the search still only looks for all shared variables - please can we look for individual references like you can for old-style globals.

Can I mention the useful help and support I got from Sarah and Ralph too. Thanks.

Cheers

Lucy

:womanvery-happy:

 

0 Kudos
Message 5 of 5
(5,705 Views)