cRIO CPU utilisation shows Time Critical at 50%, but no VIs set to Time Critical ?

[AndyClegg]

We've a live cRIO (9048, RT Liunx) running an LV2019 RT application (headless, continuously operating) and have recently started to see some excessive timing jitter in the logged data (we have started to log iteration duration of timed loop as well as CPU utilisation). The application is largely data acquisition (40msec and 1msec via DMA buffer), monitoring, slow operating logic (state machine, connected to remote UI when user needs to look at what it is doing / interact), logging data and events to a fle.It has been fairly stably operating for about a year, and is probably goes for about 1-2 months between us doing deployment of some code update that has been requested.

However, recently the some of the logged data had evidence of excessive timing jitter (dropped data, DMA buffer full messages, etc).

The cRIO CPU utilisation is normally very low (7-8%) on both cores. However, we can see periods where the CPU utilisation on Core 1 goes up to 50% for about 1.5minutes, before dropping back to 7-8%, and which seems to repeat every 10-15 minutes. This high CPU utilisation is when we see the Timed Loop itration duration go "jittery" and can be >40msec that is set as the loop time. We don't have anything within the code that operates every 10-15minutes (there is a large data file close/move that happens every 5mins, which gives a very short spike in CPU). The attached trend show the very clear changes to CPU. Normally I'd be only worried about the CPU loading if it was closer to 70-80%.

The cause of why the CPU follows this trend is not known. Even more curious (and a possible clue?) is that when the CPU does go to 50%, that 50% is almost entirely assigned as "Time Critical", even though we don't have any VIs in the application configured as Time Critical (most are set to normal priority, a couple of very simple, non-blocking VIs are set to subroutine).

So just wondering, do any of the RTOS things going on in the backrgound (Scan Engine, networking, etc) come under "Time Critical" in the CPU utilisation, or if this is actually the Timed Loops being elevated automatically by the RTOS Scheduler when it sees there are timing issues (Finished Late?).

Thanks

[AndyClegg]

BTW#1 - this is a hard to physically access system (operating at sea, unmanned) which limits what we can try, hence the focus on trying to understand what is happening through observation/data before making changes.

[AndyClegg]

Not sure if my question is either obvious, something nobody has seen before or too many words ..... I'll try something simpler.

Why could the CPU on LinuxRT cRIO show 50% Time Critical utilisation, when there are no Time Critical vis in the project ?

And here's a video showing what I mean by physically hard to access - enjoy:
https://www.youtube.com/watch?v=fbhyaiGvOLM