1473R DMA timeout at high speed?

[fineabc]

I'm using a 1473R with a Basler line camera spl2048-140km. I managed to modify the example to run the camera on both 8 tap 8 bit mode and 4 tap 12bit mode. Both run well at low speed. However, there are two issues when I try to run it at high speed.

If I set the frame trigger at relatively high speed before I start the acquiring (like 30kHz), a DMA timeout error will jump out as soon as I start it (error -8999). If I set it at low speed (like 100Hz), it will run fine as I increase it to 30kHz after it starts.

Another issue is, when I try to push the speed of the camera to its limit, which is 70kHz for single channel mode and 140kHz for dual channel mode, I will always have the error -8999. It happens when the speed reaches 100kHz in 8 tap 8 bit mode, and 50kHz at 4 tap 12 bit mode. I wonder where the bottle neck is and if there will be a way to solve it.

Thanks!

[fineabc]

Update: I tried to use two host DMAs in sequence to lower the writing and reading speed on each of them. The maximum speed improved to about 67k, quite close to what I want (70k). However, when I try to use 4 of them, the maximum speed dropped back to 50k again. The first issue is there all the time. Tried to play with the requested depth of the DMAs but did not seem to affect on the result at all.

[BlueCheese]

Just some ideas...

-Have you tried changing the DMA FIFO size larger?

-Are you using the example with DRAM buffering? At high speeds if you are going straight into host memory without a FIFO in DRAM or block ram you might have too many interruptions in accessing the PCI bus.

[fineabc]

BlueCheese,

I have tried to increase the FIFO size but did not seem to help.

About the DRAM option, I thought about it before, and looked into the vi. It seems like it just pack the data, send into a DRAM buffer, then unpack it and send it via the same DMA port. Do you think this will help to decreas the interrupts on the PCIe bus? I will try it and let everyone know how it works.

Thank you!

Zhihao

[BlueCheese]

Hi Zhihao,

The non-FPGA-enabled equivalent card (the PCIe 1433) uses a DRAM buffer to ensure no data is lost when you get to high-speed streaming data rates like you are pushing into.

What you are running into is that the code is trying to send data across the PCIe bus on every new pixel clock in your loop. The PCIe bandwidth (even though it is point-to-point) eventually has to compete with the CPU and other devices for access to memory bandwidth to DRAM. At some point the pauses on the PCIe bus bandwidth compared to how fast the pixels are coming in is going to be too high without a buffer. While the average bandwidth over the PCIe connection is fast enough for the rates you are pushing, it is not guaranteed for smaller periods of time. This is where the DRAM buffer comes in.

Eric

[fineabc]

Eric,

Thank you for the explanation. I tried the DRAM buffer method, and the speed was successfully boosted to ~90kHz from <50kHz. Also the first issue was solved. Amazing!

Now I'm trying to push the system to the limit, which is 140kHz, and I'm trying to play the old trick of using two local FIFOs in sequence. I am eager to get some further suggestions from you! 

Thank you so much!

Zhihao

[fineabc]

Tried the dual FIFO method, did not help at all. Waiting for some new ideas. Thanks!

[BlueCheese]

I'm not that familiar with the FPGA code for the 1473R, but I am fairly sure that the DRAM example has been tested with fairy fast cameras. That said, it seems from looking at it that it does not have a good feedback mechanism if PCIe bandwidth is not available. It's only feedback seems to be if the host DMA buffer fills up (e.g. host code doesn't read it fast enough). Since PCIe/memory bandwidth is extremely host-dependent, it is quite possible this example is not well tested with the combination of very fast cameras and some systems that have more jittery PCIe bandwidth. I'm not sure of the specifics of the DMA FIFO behavior, but my assumption is that the number of elements remaining really only concerns itself with host space available. For most situations, the minimal amount of FIFO built into the DMA engine likely makes this work fine.

First, I'd try instrumenting the code to figure out which specific part in the FPGA code is timing out. The example doesnt seem to track all the error cases well.

Assuming it is the host DMA that times out AND it always reports that there is free space available, my suggestion would be to modify the code to let the DMA step retry with the same data the next clock cycle on a Timeout and let the data back-up in DRAM as needed.

Eric

[fineabc]

Eric

That is some very professional suggestion. I'm busy playing with the other parts of the acquiring system recently. I'll get back to this problem as soon as I have time. I'll let you know how it works.

Thanks again for your help!

Zhihao

[C.K.Lee]

I am newbie of Labview FPGA module. I utilize PCIe-1473R to communicate with Basler spl8192-70km. Labview 1473R examples don't work. Would you give me some directions or examples to start the configuartion and capture images? Thank you.

Cheng-Kuang