cRIO CPU usage high - i801 SMBus interrupt

[Corey.Rotunno]

Hi, 

  I have been running into high CPU usage on a cRIO-9048. Even with no RT VI's running, I am often in the 30's of percents. I have done a little digging, it seems that the i801 SMBus is firing lots of interrupts. This interrupt is consistently consuming 20-25%, sometimes even more. 

Does anyone know what this is for? I am considering disabling interrupts for this. However, I would really love to know what it's doing before I go and disable interrupts. 

EDIT : I did have an NI9870 and NI9871 (serial interfaces) configured to use "Real-Time Scan" so they appear as regular COM ports. I switched this back to "LabVIEW FPGA" thinking that maybe this was the source of the interrupts. It is not 

Thanks,
Corey

[Corey.Rotunno]

Still digging into this. It seems that it is a bug specific to the Atom processor in the cRIO-9048.

This issue has been resolved as of Linux kernel 5.12. Unfortunately I am on 5.10 that comes with firmware version "8.8.0f0". Does anyone know if there are updated versions of the "NI Linux Real-Time x64" OS that I can download from somewhere? I tried downloading the latest version of "NI Linux RT System Image", but I do not see a newer firmware included with that that I can install using NI-MAX.

Any help would be greatly appreciated. 

If anyone is interested in learning more about this bug read the following : https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.11/+bug/1931001

Thanks,
Corey

[Corey.Rotunno · Accepted solution]

Feels weird to mark my own post as a solution, but I did find solution if anyone ever has same issue.

I tried downloading and installing 2023Q4 cRIO drivers .... same 5.10 kernel.

Opened NI support ticket and they responded with two possible fixes.

1. Update BIOS on cRIO. This is not easy to do as the USB ports on PC's in my enterprise are quite restrictive, so I can't make a bootable drive.
2. Instruct modprobe to ignore i801_smbus interrupt. This worked.

Run the following line over SSH:
echo "options i2c-i801 disable_features=0x10" >> /etc/modprobe.d/i2c-i801.conf && reboot.

[AndyClegg]

Thanks for sharing your problem and fix - I am sure there will be one other person who this will save a lot of time / head scratching.

[AndyClegg]

BTW - what what the symptom in terms of the CPU. You said that it was up in 30s, but was that constant, or just intermittent or growing ?

We have some weird CPU usage pattern on a deployed cRIO, where it sits at 8-10% for most of time, and then jumps up to 50% every 10-15mins and then returns to the normal low level. Nothing in the code operates at that sort of schedule, and we are looking at a firmware update when we can next get access to it (its in the middle of the sea)

[Corey.Rotunno]

Hi Andy,

  It did not really "grow", but it was variable. I saw sometimes 30%, sometimes 60 or 70%. But, the interrupt itself seemed to pretty consistently be in the mid 20's. Now that I have cleared that up, usage is pretty much as you say, sometimes in the single digits of percent, every now and then 50%. I am not sure what these spikes are exactly, but I do see that sometimes the "Web Services" spikes to around 7-8%. That is ok with me, I'll disable it if I have an issue. 

  I will say - I don't know how much I trust the usage as reported by NI MAX. My NI MAX is reporting 20 something % right now, but when I SSH in and run 'top', I get very low utilization, less than 2%. Not sure why this might be different than what max reports.

[AndyClegg]

Thanks for the reply - certainly has some similarity, and having a look at interrupts / firmware / CPU usage via Linux comamnd you mentioned seem like things to look deeper at even if to eliminate, but have to wait until we can get hands-on. We can remotely access over 4G, but a firmware update is scheduled when it next brough back to shore.

[Corey.Rotunno]

Can't believe I missed this, had the answer spelled our really well : 

https://knowledge.ni.com/KnowledgeArticleDetails?id=kA03q000000gNS0CAM&l=en-US

For some reason I found it this evening, at home, while I was googling trying to find information about the difference between CPU utilization reported by NI MAX vs. running 'top'