CAN taking Near 100% of CPU

[foreshadow]

I have been on this problem for a few weeks now and still unable to fix the problem:

Description of the problem:

1- When reading from 2 CAN channels with a Real-Time PXI Controller, the Processor runs between 70% and 80% (but most often at 80%). (Nothing else runs on the RT, but it will be required that more vis to run on it in the futur.) When running the vi on the Host PC, the processor on the RT controller tops at 100%. Some Data loss occurs.

2- Minor problem:  Board doesn't always reset even with 2 resets.)

 Program Content (view attachement for more details):

 1- While Loop at 2500us containing a sequence structure to read 2 Canbus ports in sequence. To read the CAN, I use a ncGetAttr (Read Entries Pending) followed with a ncReadNetMult.v. Data is then unbundled and set into an array in order to use a shared variable (using Real-Time FIFO). This alone takes 80% of the CPU LOAD.

2- First sequence: Canbus Reset.vi. Second Sequence has the Config (set at 1MHz), Open then CActions of Reset and Start.

Description of Task:

Read 2 CAN channels from a UUT on a Real-Time PXI Controller (8196). 16 CAN frames per whole CAN Message per channel. The 16 CAN frames are sent in 2.5ms. The Host PC that reads the data from the Real-Time Controller at any rate (I usually set it at 4 times the loop time of the RT Target).

FIFO Configuration:

12 elements of the CAN frame + RT TickCounter. As 64bits Double or 64 bits U64. 13 elements per array. 512 Arrays (this number varied alot). The lower the # of arrays, the more random data loss occurs, but the higher it is, the more the duration of data loss is long. Note that sometimes i get the last frame and appears in the log as many times as loop size.

Host PC:

Reads the Shared Variable in a For Loop with the #the same as the FIFO size (or higher). 

Note: Many Disable diagrams, but doesn't affect the program in any way. Its other tests done.

Thank you. 

[Caleb_W]

Hello,

Is there any way that you would be able to remove writing to the shared variable for testing purposes?  I would like to find more on where the bottleneck is coming from so I can get a better idea on what can be done to solve your problem. 

---

 1- While Loop at 2500us containing a sequence structure to read 2 Canbus ports in sequence. To read the CAN, I use a ncGetAttr (Read Entries Pending) followed with a ncReadNetMult.v. Data is then unbundled and set into an array in order to use a shared variable (using Real-Time FIFO). This alone takes 80% of the CPU LOAD.

---

Do you mean that unbundling and setting the shared variable takes 80% or everything you mentioned there takes 80% of the CPU?

[foreshadow]

hello, the CAN alone takes 80% of the CPU Load. In other words, (ncGetAttr + ncReadNetMult) x2 Channels = 80%.

[O_Proulx]

Hello foreshadow,

I think that the processing of the frames after they are read in is taking a lot of processing time.  You are dynamically building arrays which takes a lot of processing and memory. You can verify this by wiring a constant of 1 in the iteration terminal of your for loops (this way, the for loop will only run once).  You should notice that your processor usage goes does.

I would suggest preallocating the array using  and then using the initialize array.vi and then using replace array subset.vi to access the array.  You can find more information about this here.  To eliminate any arrays handling, you could also use a seperate variable for all the frame's parameters. 

Let me know what the results of these tests are.

Have a great week.

[Caleb_W]

Hi,

How fast are you expecting these CAN messages to come through.  I see that you are running each iteration at 2.5 ms.  Also, I was wondering if you could strip down the application where you are not building or indexing any arrays and simply reading and displaying the CAN messages.  Most of the processing for the CAN messages is done on the hardware.  Which CAN card are you using for this application?

[foreshadow]

Thanks for the replies, I fixed the loss of data, but not the CPU load by doing the following:

I finally tried Queues on the RT (didn't earlier because it was way too slow to execute when I was running my simulation on Windows). I based my vi with the NI Example Queue basics.vi. I also optimized CReadMult, I send the Data direclty into an array with I insert the Tickcount* then send it to the Host PC for it to process the rest of the frame building and then to process the information. Doing both (Queues + CReadMult) helped since now I do not have anymore loss of data, however the CPU LOAD goes up to from ~75% (at start) to 87-89% and stays there.

*Tickcount: I've noticed a weird effect with the Tickcount: I'm sending the msec time and the loop which adds it to the array is set at 2.5ms, however the tickcount read at the Host PC is always on a 10ms timer difference (instead of ~2.5) which also seems to correspond to the same loop timer as the secondary while loops that all they do is sent the information to the Host PC.  The priority of the CAN loop is 1 and the priority of the SV loops are 100. Unless i'm mistaken, priority 1 should be of highest priority and 100 is lowest (excuted only when CPU has "extra" time).

With the Queues: 

Currently it worked (without data loss) when both CAN channels are read in sequence with 2 different loops to send the informatiosn to the PC through shared variables. The RT FIFO is set at 32 Arrays of 2048 Elements. 2048 because the CReadMult seemed to have been going up really high randomly. I have not had the time to optimize the RT FIFO or really know how to cound it exactly. When using the size indicator of the Queues, they are always full. I only use Enqueue and Dequeue.

@O_Proulx:

CReadMult seems to be the main problem of the dynamic array since the information gotten from the CAN seems to vary alot. Are you saying I should insert the data from CReadMult into a fixed array size before sending it through the SV ? I don't think that would help since it seems that the CReadMult automatically creates dynamic arrays.

 @Caleb:

Like I said previously in this post, I stripped everything down, even the bitwise operations of CReadMult to do them on the Host PC. I'm using the NI PXI-8461/2 Series 2 2-Port High-Speed Can. Next time I get time on the benchtest, I'll verify the CPU Load time when removing the SV loops.

Thanks. 

[foreshadow]

Ok I re-run the tests on a very short time (few seconds to a few minutes).

Sequence or Parellel reading of CAN seems to have the same performance.

CAN alone with Queue release added w/o SV: 49%CPU LOAD.

CAN w/ SV: 52-53%CPU LOAD (initial burst > 78%).

CAN w/ SV and Interface on Host PC running: 62%CPU LOAD.

I was able to get the Benchtest for only a few minutes, so I wasn't able to run the program for long.

Isn't 62% CPU LOAD high for a simple reading of 2 CAN ports with sending the informations to the Host PC? We have a low CPU LOAD when using ARINC to read and write with the RT.

I still had the tickcount problem which didn't take the msec when called in the 2.5s loops.

[Caleb_W]

It looks like you are doing a loop of 2.5 ms and not 2.5 s.  Are you trying to check the information that quickly?  The dt is based off of the clock you have specified which is 1 MHz.  2500 ticks of the 1 MHz clock will get you a loop time of 2.5 ms which could significantly raise your CPU usage if you are only expecting to be checking every 2.5 seconds.

[foreshadow]

The loops are 2.5ms. I made a misstype, sorry for the confusion.

[O_Proulx]

Hello foreshadow,

It's good to hear that you were able to get the CPU down a little bit. 

In answer to the dynamic array question, the NI-CAN driver reserves some memory space for the read frames.  So it does not have have to reserve memory dynamically, in comparison to building arrays in LabVIEW, where LabVIEW has to allocate and then deallocate memory in the loop.

Another option we can try to reduce the CPU usage would be to wait until there are 16 frames in the buffer.  So, instead of reading when the are more that 0 frames, we could wait until there are the 16 wanted frames in the buffer (go into the case structure when available frame >= 16).

Apart from that, it is possible that the board is taking a lot of the CPU time, since it uses interrupts to indicate that frames are available.  The NI-CAN driver should manage those interrupts effectively if there are more processes running.  Please keep us posted if you run into issues in the future, when you have other tasks working in parrallel to NI-CAN.

Have a great day.