nct ReadTimestamped returns error bff6211a

[VIGuy]

We have an CAN monitoring application that runs on a few different platforms (desktops with NI PCI CAN cards and laptops with NI PCMCIA CAN cards).  The application uses the Channel API.  We are using NI-CAN 2.5.2f0.   We monitor 32 CAN Channels spread over three different Frames (2 16-bit Channels in Frame 1, 2 16-bit channels in Frame 2, and 28 1-bit channels in Frame 3).  Each Frame is broadcast at 200 Hz.  Each Channel is monitored in its own task.

After a seemingly random amount of time (a few hours to a few days) all platforms will eventually throw a 0xBFF6211A.  For those who don't speak error code here is the description:

"NI-CAN:  (Hex 0xBFF6211A) NI-CAN copies received CAN frames from the card to a large kernel-level queue.  There is one queue for each Channel API message and one queue for each Frame API Network Interface. If your application does not read fast enough to empty this queue over time, the newest frames are lost.  Solutions: Call Read more frequently (such as by reducing user-interface code); Frame API: Use filter attributes to reduce receive traffic; Channel API: Reduce the number of messages in your tasks; Use single sample Input (sample rate 0)."

Given that input, I check scan rates and reading rates and all that and couldn't find anything.  Finally I set up NI-SPY and ran it until it failed.  The NI SPY log tells me nctReadTimestamped for Task ID 1 is called pretty consistently every 370 mS (varies some from 330 to 390 -- no reading exceeded 390 mS).  Typically the call returns 73, 74 or 75 updates.  The math says that's to be expected (200 Hz = 5 mS, 370 / 5 = 74).  The time between the last successful call and the call that failed was 330 mS.  The call that failed did return 15 updates.

The NI SPY log clearly shows I'm servicing the Read in a timely manner.  The NI SPY log clearly shows the queue isn't backing up.  The broadcast rate isn't excessive.  So what's going on?  All help appreciated.

[DirkW]

Hi

What is the mode you are running the monitoring Tasks in? (Timestamped Input?)

Could you share the Spy log file with us?

DirkW

[VIGuy]

Hello Dirk,
Yes the mode is Timestamped Input.

SPY log attached.
Also, included is a VI to help parse the log file.



Message Edited by VI Guy on 03-20-2008 12:49 PM

Message Edited by VI Guy on 03-20-2008 12:51 PM

[DirkW]

Hi,

It seems that having 28 tasks for the same message somehow causes a problem, that when the driver trys to read the first task again it runs into a overflow situation.

Is this allways happen on Task 1?

In your initial post you mentioned that you have 32 different channels in 3 messages. Why shows the log file 33 tasks then?

If you configure a task for every channel in the message the driver internally copies every received message as a copy to every task queue. That means if you have 28 tasks for the same message and you receive a single message the driver creates 28 copies for this message. It seems that the driver is busy doing that and your first task overflows.

Because this happens in the background you do not have any influence on it. It could be caused whenever the driver has problems to allocate memory fast enough. (Fragmentation, other windows tasks)

You have two options here.

The first and easiest would be to modify the queue size for the overall receive memory. That could help. At least you could try it.

Go to your National Instruments/NI-CAN/bin folder and open the nicanopt.ini file. Within the [Internal Channel Queue] Section change the value for "NI_RQL=" to 150 for example.

Perhaps that helps to avoid the overflow.

The second option is to create one Task per message. If you read timestamped, thats the most performant way to do receive your data.

The bad thing is that you have to deal with the two dimensional arrays of cluster. ;-(

But the best solution would be the one task per message option because it fixes the root cause.

DirkW

[VIGuy]

Hello Dirk,

It is unknown if it always happens on Task 1.

The typical full sized CAN frame has what, 16 to 20 bytes of data? It appears you are suggesting the NI-CAN driver is unable to copy 20 bytes of data 28 times in 5 mS?  I am skeptical this is the root problem.

The SPY log show the queue is not backing up.  Each call requests 1000 updates.  The call always returns the proper number of updates for the time elapsed between calls.  The number of returned updates is not steadily increasing, as would happen if the queue was backing up.

I do see the default value for NI_RQL is 80.  If that number is the number of updates to hold before throwing a 0xBFF6211A, it is plausible there lies the problem as we are operating near that limit.  I will increase this value to 150 as suggested.

[VIGuy]

We had another instance of the monitor (different .exe version as well as different hardware setup) fail today.  SPY file shows us reading at 200 mS intervals, returning 40 to 45 updates per read.  On this machine NI-RQL=80.  Its task ID 15 that fails, but its the first task that actually has any data being transmitted.