Hi there! (sorry for the long description and for the lack of attached VIs, no permission here)
I'm using LabVIEW 2019 (32-bit) and a cRIO-9049 to implement a XYZ motion control system, which must follow trajectory references. Closed loop control (feedback readings + control laws calculation + actuators commands) is implemented in FPGA at 20 kHz (50 µs period). Trajectory references discretized at 20 kHz are read from a single CSV file by RT and streamed to FPGA using a single Host to Target DMA FIFO channel.
The problem: sometimes I get a read timeout on FPGA side, and I just can't explain why, neither solve it. For a same trajecory file, this error occurs roughly in 4 of 10 attempts, so most of time it runs without problems, but it eventually fails at some random sample of the trajectory, no regularity observed.
I can't ignore this error by waiting until a new sample arrives on FPGA side, because this would distort the motion profile.
A "trajectory sample" consists of 8 elements (SGL type) :
- 1 trigger (irrelevant to this discussion)
- 3 position references (X, Y and Z)
- 3 feedforward control efforts (X, Y and Z)
- 1 dummy element, to make total size a power of 2 (required by the "Number of elements per read" FIFO setting)
As suggested here and here, to make an efficient usage of DMA FIFO, I write the data by blocks, which size is configurable before starting the streaming. I've been mostly working with a 200 ms block size, which corresponds to (200 ms/block)*(20 kHz)*(8 elements/sample) = 32000 elements/block, although I tested others sizes, and didn't solve the problem.
FPGA consumes these blocks sample-wise (8 elements per reading) at the 20 kHz controller rate. On RT side, I implement a loop to send a block per iteration at a rate equal to the block duration. This way, I try to keep the "mean flow of writes" equal to the "mean flow of reads". For example, with 200 ms block size, my RT loop runs with a 200 ms iteration loop time. To give some advantage for the RT, I only start reading blocks in FPGA after writing two blocks, i.e., I enable FPGA FIFO reading (using a FPGA front-panel boolean) only at the end of the second iteration from the RT loop.
I've tried different implementations of this RT loop, all of which I'm pretty confident can run every iteration faster than the specified loop time/block duration.
First, I tried different loop structures and timing monitoring, such as:
- While loop timed with Wait Until Next ms Multiple and monitored using different Tick Count's and Flat Sequences
- Timed loop, monitoring with Tick Counts and left node readings, like Finished Late? and Iteration Duration
- Timed loop set with dedicated CPU, monitoring with Tick Counts and left node readings, like Finished Late? and Iteration Duration
For these options, I've tried:
- Reading each block from the CSV file inside each iteration of the RT loop, which is less recommended due to File I/O overhead/indeterminism, but still didn't result in any "late iteration"
- Reading the entire file beforehand and then start the loop, which uses a lot of memory in RT, but drastically reduces execution time of iterations
For a 200 ms block size, the first approach resulted in iteration of ~130 ms, and the second approach around 200 us. FIFO read timeout in FPGA occured in all combinations of the two previous lists.
I've also played with buffer sizes. In RT it's depth is set to 160768, corresponding to ~1 second of trajectory (8 elements/sample * 20 kHz), and I never got even close to fill it. In FPGA, I've tried the following sizes:
- 1029 (minimum configurable, recommended here)
- 32815 (enough for 1 block of 200 ms)
- 65583 (enough for 2 blocks of 200 ms)
Other things I've tried, without success:
- Set different timeouts to the FPGA FIFO Read Method, from 0 to 40 ticks (1 us, for 40 MHz FPGA clock), above which would start to compromise my closed-loop implementation
- Set Number of Elements Per Read in FIFO configuration to 1 (instead of 8), and run FIFO Read method inside a For Loop 8 times
- Start reading FIFO in FPGA after the third block was written by RT, instead of the second
Since all iterations of RT loop seem to run on time (based on tick counter and timed loops left node readings) and is always at least one block ahead from FPGA, I started to suspect on the FPGA side or the DMA FIFO controller itself.
I started monitoring both FIFOs using "Empty Elements Remaining" (RT) and "Get Number of Elements to Read" (FPGA) Invoke Methods and noticed that:
- When no problem occurs, they indicates the number of elements on both FIFOs remains roughly constant, indicating the proper balance between flows of writes and reads, as expected. In RT's FIFO, it keeps close to 64000 (two blocks of 200 ms), and in FPGA's FIFO, close to its configured size;
- A couple of seconds before a read timeout occurs in FPGA, the number of elements in RT side starts to fall at a rate of ~2 kHz, while the number of elements in FPGA FIFO remains roughly the same, until the RT FIFO is empty. At this moment, the number of elements in FPGA FIFO starts to fall, at a rate of 22 kHz, which is 2 kHz above the excepted 20 kHz rate
I've checked this 2 kHz and 22 kHz by reading the Number of Elements in both FIFOs periodically using different methods in RT and post-analysis, and it really seems that the FPGA suddenly increases the rate at which it reads the FIFO. This could explain why I get the timeout, but then I have no explanation for how this could occur.
I doubled-checked (by using, for example, a dedicated Timed Loop and monitoring its Iteration Durations) whether this sudden rate increase could be just a problem in the RT implementation of this measurement, and I'm confident it's not a false conclusion.
The FPGA loop responsible for reading the FIFO at 20 kHz is implemented simply using a While loop, a Flat Sequence and a Loop Timer with a constant input, as recommended in many examples. I tried configuring this Loop Timer by ticks and by µs, no difference observed. And when I check the difference of its output between each iteration, it just stays constant and equal to its input value, so no indication it increased the loop rate.
Lastly, both implementations in RT and FPGA have a lot going on besides this data streaming, and we consider this could also explain the problem, although all loops had their iteration duration checked, CPU loads in RT are fine, and the most used resources in FPGA are slices (71%) and LUTs (52%).
In summary:
- In some attempts to stream data from Host (RT) to Target (FPGA) using a DMA FIFO, I get read timeout on FPGA side;
- I can't ignore it or wait for a new sample to arrive, cause it would compromise motion trajectories;
- Data is sent periodically by chuncks, to improve streaming efficiency. Different implementations of RT streaming loop were tested and all respect iterations deadline, in order to neither fill, or empty FIFO buffer;
- Different configurations of FIFO sizes, timeouts and loop timings were tested, but none solved the problem;
- RT streaming is always at least 2 chuncks ahead from FPGA reading (except when the problem occurs);
- I have evidence (checked using different approaches) that the timeout occurs because FPGA suddenly starts to read FIFO faster that expected (22 kHz, instead 20 kHz, from which 2 kHz difference is the same rate RT FIFO starts to empty before a read timeout occurs);
- Application is running a lot of stuff in parallel both in RT and FPGA, although proper loop timings were repeatedly confirmed by different methods
I'm stuck with this problem for at least 2 weeks already, and need to get a solution as soon as possible. If anyone have any ideas of something I'm missing new tests I could try, I would really appreaciate! Again, I wish I could post some VI's, but at the moment I have no permission.
Best Regards,
Gabriel O. Brunheira
Mechatronics Engineer
Brazilian Synchrotron Light Source
