CRIO network stop working for no apparent reason

[huotj]

Hello, 

We use a CRIO-9074 to read weather sensors. The code works well, but around once per day, the network communication stop ; we are not able to ping it and the tcp/ip communication stop. If we reboot the device the network work and even if we do not touch anything, sometimes the network start working again. I know, with the logs, that the CRIO continue to work normally and gather data as planned, but it cannot be accessed by network. 

We found that problem on november 18 after years of working with the same code without problems. 4 days after the first network shutdown, the crio stopped work at all. We send it to repair  and they return it and they said they replace a board of DC offset signal or something of the sort (it was translated to me).  We receive it last week and try it out, it worked, but the same network shutdown continue to occur. 

I'm afraid that maybe they just repair the symptoms and the real cause is still there. Anybody have an idea about it ? 

Thank you, 

Julien 

[ARVega]

This kind of sounds like starvation. 

Could you monitor the CPU usage during the time it does not communicate in the network? in case that it was not visible you could create a function that would write a time stamp and the value of CPU usage in a txt file every couple of seconds. 

If the cRIO is starving, you would see a peak during the network shutdown time or even you could see that no data was written during that time...

[huotj]

Hello, 

Thank you for taking the time, 

I wrote in a file every minute the cpu and memory and they are stable through all the day, even during the 2-4 hour that the network do not work. So if I understand starvation enough, that's mean it's not starvation ? 

What is strange though, is that the network failure happens twice a day and one of them is always around 0700 AM. 

Julien 

[Matthew_Williams]

Hi Julien.  Years of proper operation followed by a distinct change in performance, particularly with a relatively consistent 0700 schedule sure sounds like an external influence.

Most of our deployed equipment is FieldPoint, but I think the -9074 might be a similar vintage OS.  We have found instances where network scanning security tools can crash the TCP stack or the controller.  Modern networks, even internal networks, are frequently not happy places for devices that can't defend themselves.  Given the sudden onset and regularity of the problem, I would also consider network causes; a security scan, large bandwidth backup from a (network-wise) nearby device, etc.

Finally, inspect the local environment.  We once had a piece of equipment at a customer facility start failing every day at about the same time.  Couldn't figure it out until I happened to be talking on an adjacent landline when it all of a sudden it went to static and then silent.  Turns out they had installed a diathermy machine immediately on the other side of the wall from the phone and our equipment.  Every afternoon when they used it, the problem appeared.  Moved the phone and our equipment 4 feet to the left, problem solved.

Good luck, hope that helps.

Matt

[huotj]

Thank you, 

I will look theses sources it's sound like external influence also to me. 

Julien