Mysterious Intermittent Spikes in Loop Time: Battling Inconsistency in Real-Time Serial Communication

[johndoe626]

Hello everyone,

I'm facing a perplexing issue with my real-time system and would greatly appreciate any insights or suggestions from the community.

I'm working with an NI-9149 expansion chassis that has 8 slots, but I'm only using one serial card in the first slot. The card is connected to two identical RS485 Modbus sensors. Each sensor receives a 10-byte request and replies with 9 bytes of data. The total read/write time for each sensor is approximately 25 ms on average.

My goal is to consistently retrieve data from both sensors every 50 ms. However, I'm encountering strange intermittent spikes in the loop time, causing it to occasionally exceed the 50 ms. For example, the loop time may reach 54 or 70 ms and then suddenly return to normal. These spikes are consistent in frequency but intermittent in time.

Observations:
- The amplitude and frequency of the spikes increase when I connect four sensors instead of two. (to 80 ms and it occurs more than before)
- The same problem occurs when I use an older compact-RIO chassis module.
- I have ruled out software or code-related issues through extensive testing, including collecting sensor readings using the chassis' FPGA instead of remote collection via VISA Server.
- Interestingly, when I replace the chassis with an Sb-RIO, the issue does not occur at all.

Suspicions:
I'm beginning to suspect that there may be limitations or constraints related to Ethernet communication for cRIO expansion chassis. It seems strange that a chassis with 8 slots would struggle to maintain consistent 50 ms loop times with just one serial card and a few RS485 sensors.

Questions:
1. Has anyone encountered similar issues with intermittent spikes in loop time when working with real-time serial communication?
2. Are there any known limitations or considerations specific to cRIO expansion chassis that could be causing this behavior?
3. Are there any recommended strategies or workarounds to mitigate these spikes and achieve consistent loop times?
4. Could the issue be related to Ethernet communication, given that it doesn't occur with an Sb-RIO?

Any thoughts, experiences, or suggestions would be greatly appreciated. I'm eager to find a solution and ensure reliable real-time performance in my system.

Thank you in advance for your help!

Best regards,


Note: I'll attach a photo that's simulating what I'm experiencing.

[ZYOng]

What you are seeing is expected. Ethernet communication is not deterministic by nature.

From Expansion I/O for LabVIEW RIO Systems: An In-Depth Comparison:

The communication jitter for MXI-Express RIO and EtherCAT RIO is less than 10 microseconds and less than one microsecond, respectively. Ethernet RIO devices do not have a specification for jitter. This is because Fast Ethernet and wireless transmission are not deterministic methods of transferring data.

[johndoe626]

Well, I know that Ethernet communication aren't very deterministic.
But still, I thought it shall be reliable for my setup and requirements.
As I already have many other Ethernet devices on the setup, Like, Sb-RIO to Another Sb-RIO (with serial card).
This issue is never occurred in that setup. Also, the other Ethernet devices is ok.
It's just the Chassis.

So I do want really to know what's the main source of the issue.
Is it Chassis' CPU?
Is it Chassis' Ethernet Interface?
Or is it another element?

Cause I do want to think about workaround, as possible as I can.