cRIO CPU % Total Load Drifting Up...

[S1ack]

My project has a cRIO 9074 in Hybrid mode (one Analog card is under the FPGA target, all others are scan engine).

The cRIO is a central controller to 3 modbus slaves (ModbusTCP function block palette [i.e. not the I/O Server]). And is a slave to 1 SCADA system via Ethernet/IP.

One of the slaves is an XP based HMI (not NI product) that is logging data to SQL DB. Which is how I get the plot shown. Weather I deploy dynamically, or 'run as startup' over the span of about 4 hours, the CPU % total load creeps upwards and rails. Eventually the HMI looses comms with the cRIO and stops logging data.

I have a feeling the culprit is the timed loops component of the total cpu load - but to confirm that, I will need to add it to the Modbus data to be logged.

The application has many timed and untimed parallel loops, some for the ModbusTCP comms, some for other logic - including a Queued state machine. During the four hours, the state machine is static - i.e. no transitions, and the size of data, as well as the polled frequency of data on ModbusTCP, and Ethernet/IP is also static. I do no programmatic changes to the data between the cRIO and the outside world.

Also, no logging to the cRIO disk. On an error, I may post a message to the console. (I'll check on that).

If this were your problem, how would you go about debugging it?

[christian_w]

Use the Distributed System Manager (DSM) to get more information, also about memory usage and the load balance between timed and untimed structures on your RT CPU.

[S1ack]

It is timed structures.

It is a memory issue (or so I would infer from what the console said)....

0x418a680 (LV TL[600]: Faults): memPartAlloc: block too big 11587600 bytes (0x8
aligned) in partition 0x32d4f8
0x418a680 (LV TL[600]: Faults): memPartAlloc: block too big 11595792 bytes (0x8
aligned) in partition 0x32d4f8

Sent source into NI. Will update this thread when I learn how I caused this.

[S1ack · Accepted solution]

per NI:

I used the system profiler tool to snapshot timing every hour or so. Smoking gun pointed to 2 vis. They were taking increasingly longer to execute, for one it's average excecuton time was 6.5ms (immediately after deployement) and 24.4 ms after about 4 hours - and it drifted up as the 'worst offender' accoding to the profler snapshot data. In this vi I was using the 'obtain queue' function - without an associated release queue. 

I hardwired the queue reference to these vis. And so far 3 hours in and my CPU Total Load %  is holding steady.