Explanation of DMA errors

[A_squared]

I'm working on some modifications to my existing PCI-6229 driver.  It uses DMA transfers to get the AI values from the card.  The card is acquiring at ~80 kS/sec and the driver is performing the DMA transfers at a rate of 4 kHz.  Every 250 usec the driver reads the number of bytes available, determines the number of bytes to read to get an equal number of values for each channel, then reads the corresponding number of bytes from the card.  This attempts to correct for jitter between the card and computer, delayed entry, etc.  Below is a snippet of the code.

//Code begin

m_status = dma->read(0, NULL, &bytesAvailable);
m_numPoints = (unsigned int) floor( (float) bytesAvailable /( 2.0 * (float) m_numInputs ) );
bytesAvailable = 2 * m_numInputs * m_numPoints;
//2 bytes per sample
//read even number of samples from each channel
if( bytesAvailable > 0 )
{
   m_status = dma->read(bytesAvailable, (u8 *)rawBuffer, &bytesAvailable);
   if( m_status != kNoError )
   {
      printf ("error: dma read (%d)\n", m_status);
      return 1;
   }

   /* convert/scale data from rawBuffer and store */
}

//Code end

The problem occurs at the second DMA read where it produces an error of 3 some small fraction of the time.  Correlating this to the enum below, I've determined that my error is "Data Not Available".

kNoError = 0,
kBufferOverflow,
kBufferUnderflow,
kDataNotAvailable,
kSpaceNotAvailable,
kWrongState,
kInvalidChannel,
kInvalidInput

This may be a stupid question, but what causes these particular errors.  Can you tell me how errors 1 and 4 are different and how errors 2 and 3 are different and specifically what causes error 3?

I'm trying to deal with some stability issues, which may or may not be the result of this problem, and so I'm currently tracking down any anomalies and determine where everything goes wrong.  Thanks.

[Tom_W_[DE]]

Hi A squared-

The error aliases are perhaps a bit obscure.  Let me see if I can offer some explanation (all of this is basically coming out of tDMAChannel.cpp)

kNoError = 0
   Self explanatory

kBufferOverflow
   Occurs when bytesInBuffer > _size (i.e. buffer size).  This means that we wrapped around the circular DMA buffer before transferring some number of samples to the application buffer, thus overflowing the DMA buffer.  The number of overwritten samples would be equal to (bytesInBuffer - _size).  This error only applies to input operations.

kBufferUnderflow
   The opposite case from kBufferOverflow, this means we did not fill an output buffer fast enough and that the DMA process transferred stale data to the device.

kDataNotAvailable
   This is the case where requestedBytes > bytesInBuffer.  This error is the motivation for performing two read operations; the first read to determine the current quantity of bytesInBuffer so that we can intelligently choose the requestedBytes value for the second read operation.

kSpaceNotAvailable
   This error is thrown when a buffer allocation fails.  Your OS Interface should return an appropriate status value after the _buffer->allocate() call in

tDMAChannel::config().

kWrongState
   This is an internal error for the DMA library that is thrown whenever the expected incoming state doesn't match a requested state change (for example, you can not read() from a kIdle state.

kInvalidChannel
   Not used.

kInvalidInput
  

Not used.

Hopefully this helps-

[A_squared]

Thanks for the quick reply and useful information.  This was as I expected.  The DMA error is telling me that I am requesting more data that is available.  If you look in the first post, you'll notice there are 2 dma->read's.  The first determines the number of bytes available and the second reads less than or equal to the number of bytes available.  In my current configuration, I should read on average 40 values (80 bytes) per loop.

The program is spitting out dma 3 error messages.  On the loops that produce the error, the program reports the following information:
first dma->read( 0, NULL, &bytesAvailable) returns 82 in bytesAvailable
second dma->read( bytesAvailable, (u8 *)rawBuffer, &bytesLeft) sends 82 in bytesAvailable, returns 7385656 in bytesLeft, and return DMA error 3 -  Data Not Available

So I shouldn't be requesting more data than is available.

This is more of a status at this point.  I'll start the real detective work now, but if anyone sees an obvious mistake please let me know.

On a slightly separate note, I have a question about DMA transfer modes, as I wasn't able to find much information on them.  The DDK ships with an example that uses the Ring mode, but there are other modes (e.g. Normal, LinkChain, and LinkChainRing).  How do these modify the transfers?

Thanks.

Aaron

[Tom_W_[DE]]

Hi Aaron-

The other modes are not exposed or available in the DDK.  In general, Ring mode is the easiest to implement and the least complex from a memory management standpoint.  Consequently, it requires the driver client to be more diligent about checking for things like overwritten samples, etc.  Ring mode literally just pumps samples into a circular buffer and keeps cycling over itself.  This situation is the motivation for the extensive error checking code in tDMAChannel::read().  The other modes don't offer any obvious performance enhancements once all the memory is allocated, and (IMO) Ring mode is definitely the easiest to work with from the perspective of complexity in programming the hardware interface.

[A_squared]

The application I'm developing is on InTime.  Tenasys supplied me with their version of the OS specific code and they reference Normal mode so that was mostly curiousity. 

I've been delving into my kDataNotAvailable error.  The error occurs approximately 0.5% of the time and doesn't occur at uniform time intervals.

The program does the initial dma->read to determine the number of bytes available.  I'm current running a loop at 1000 Hz with a low level timer and the DAQ card is acquiring 2 analog inputs at a rate of 10,000 Hz; therefore, 40 bytes should be available at each loop.  The initial dma->read reports 40 bytes.  I'm currently feeding this number directly to the second dma-read.  The second dma-read reports the kDataNotAvailable error and now has a printf statement to report the requestedBytes and the bytesInBuffer.  The requestedBytes is obviously the 40 bytes returned by the first dma->read, but the bytesInBuffer is only 38 now.  The problem seems to suggest _getBytesInBuffer function in tDMAChannel.cpp is the culprit.  Why this _getBytesInBuffer function is returning a smaller number if no data has been read is the big question.  I'll try to determine if the read index or write index is the one changing next, but if there is a known bug in this function please let me know.  Is it possible the _miteChannel->DeviceAddress.readRegister() - _miteChannel->FifoCount.readFifoCR() should ever return a smaller number besides rolling over?  Thanks

[A_squared]

I'm following _readIdx, _writeIdx, _miteChannel->DeviceAddress.readRegister (), _miteChannel->FifoCount.readFifoCR(), and bytesInBuffer in the tDMAChannel class.  In the short time between the first dma-read and the second, everything remains constant except _miteChannel->FifoCount.readFifoCR() increases from zero to two.  This causes _writeIdx to be decremented by two ( _writeIdx -= _miteChannel->FifoCount.readFifoCR(); ) during the second dma->read.  Can someone explain to me what the readFifoCR function does and is returning?  Without knowing the exact roll of readFifoCR but assuming the code is correct, my thought now is that maybe both readRegister and readFifoCR increment between the readRegister and readFifoCR lines resulting in an outdated value in _writeIdx.  Thanks.

[A_squared]

I shouldn't have done so much thinking out loud and just did it because it was the obvious answer.  Indeed, it appears that sometimes readRegister and readFifoCR both increment after readRegister executes and before readFifoCR executes.

Here is my band-aid:

_writeIdx = _miteChannel->DeviceAddress.readRegister() - _miteChannel->FifoCount.readFifoCR();
tempIdx = _miteChannel->DeviceAddress.readRegister() - _miteChannel->FifoCount.readFifoCR();
if( tempIdx > _writeIdx )
    _writeIdx = tempIdx;

This way if the readRegister and readFifoCR increment in the first line, it will hopefully operate fast enough to not occur on the second line.  There may be beter ways to do this and my primitive understanding of the function of these functions may prevent me from developing a better solution, so I would appreciate an expert's opinion.  I'm also not sure of the performance hit of this although I would expect it wouldn't be too severe.

Aaron

Note:  With the above code, I am not observing any further dma errors and I've even upped the loop rate to 2000 Hz and the acquisition rate to 160kHz on 2 analog inputs.  Because of the time it takes to get from class constructor to the time when the infinite loop starts acquiring data from the card, the buffer overflows (kBufferOverFlow error occurs).  I only want this to be a one time error, so I modified the kBufferOverFlow code in tDMAChannel.cpp to be:

if( bytesInBuffer > _size )
{
    _readIdx = _writeIdx;
    return kBufferOverflow;
}

Message Edited by A squared on 06-21-2007 06:12 PM

[Tom_W_[DE]]

Hi Aaron-

Good idea- making sure that the data is stable (or at least valid) is certainly helpful.  Your method seems to be working relatively well.  I did some digging into the mite control code that we use for a different product.  This method also uses two reads of the DeviceAddress Register (aka DAR, please excuse the pseudo-code):

tBoolean darStabilized;
u32 numTries = 0;
u32 firstDAR, secondDAR;

do {

   firstDAR = _readDAR();
   fifoCount = _readFCR();
   secondDAR = _readDAR();  

   darStabilized = (firstDAR == secondDAR);

   ++numTries;

   }

while ( !(darStabilized && (secondDAR >= fifoCount))

               && (secondDAR <= (fifoCount + ( secondDAR - firstDAR )))

               && (numTries < 100));

secondDAR -= ( darStabilized ) ? fifoCount : (fifoCount + ( secondDAR - firstDAR ));

secondDAR is then analogous to the _bytesAvailable value in the MHDDK.

To give you a bit more of an idea what is going on here, the hardware uses the DAR to represent the total number of samples that have been acquired and queued for transfer to memory.  The FIFOCount Register (FCR) gives an indication of how many samples are still queued (i.e. that haven't been transfered to memory yet).  So, the difference between the two is obviously what should be available to be pulled into your application from RAM.  The challenge is making sure that the two readings are in sync as both are constantly changing (presumably DAR will always be increasing while, assuming good bus throughput to memory, the FCR should remain relatively low.

I have seen this error inexplicably very rarely when testing some things here.  It sounds like your system is particularly susceptible to getting out of sync with these readings.  I would encourage you to give the above implementation a try since it has proven to be very reliable in the area in which it is used.  Any feedback about how it works for you would be very valuable and appreciated.

Thanks-

Message Edited by Tom W [DE] on 06-21-2007 06:33 PM

[A_squared]

Thank you, I will adapt the code you provided for the M series card I have.

Additionally, I haven't seen an error with this yet, but I wonder if one will pop up.  The code I provided to reset the buffer overflow just sets the read index to the write index.  I did this because I observed the write index to always increment by the number of analog inputs being scanned.  Is it possible that this code could lose track of which input is which?  Thanks.

[Tom_W_[DE]]

Hey Aaron-

In case it helps, here's a snippet from the changes I'm testing with for the _getBytesInBuffer() method:

u32 tDMAChannel::_getBytesInBuffer ()
{

// 1. Update MITE's location

if (_direction == kIn)

{  
  

//
   // the device address indicates the number of bytes that have
   // been transfered from the device to the DMA FIFO.
   // Samples in the FIFO have not been transfered to host memory
   // so correct the number device address with the FIFO count.
   //
   // The loop avoids a _writeIdx underflows.
   //
   // 'tries' is there to have an upper bound, but it should never
   // exceed 100.
   //

   u32 deviceAddress = 0;
   u32 deviceAddress1 = 0;
   u32 fifoCount = 0;
   u32 tries = 0;
   tBoolean stableDAR;

   do
   {
     deviceAddress = _miteChannel->DeviceAddress.readRegister ();
     fifoCount = _miteChannel->FifoCount.readFifoCR();
     deviceAddress1 = _miteChannel->DeviceAddress.readRegister ();

     stableDAR = (deviceAddress == deviceAddress1);

     ++tries;
   } while ( !(stableDAR && (deviceAddress1 >= fifoCount))
              && (deviceAddress1 <= (fifoCount + ( deviceAddress1 - deviceAddress )))
              && (tries < 100));

   deviceAddress1 -= ( stableDAR ) ? fifoCount : (fifoCount + ( deviceAddress1 - deviceAddress ));
   _writeIdx = deviceAddress1;
}

else
{
   _readIdx = _miteChannel->DeviceAddress.readRegister ();
}

// 2. Calculate difference between read and write indexes
// checking for rollovers

u32 bytesInBuffer = 0;

if ( _writeIdx < _readIdx )
{
   bytesInBuffer = 0xffffffff - ( _readIdx - _writeIdx );
   ++bytesInBuffer;
} else
{
   bytesInBuffer = _writeIdx - _readIdx;
} return bytesInBuffer;
}

Does this implementation work better in your testing?  Any other unforseen problems?  I'm trying to nail down the best way to show this in the MHDDK, so any feedback you have would be appreciated.

Thanks-