Real-Time Measurement and Control

cancel
Showing results for 
Search instead for 
Did you mean: 

DMA unstability problem at FPGA start

Hi everyone.
We are currently migrating a condition monitoring application from PXI to CompactRIO. For that, we obviously have to re-program data acquisition from NI-DAQmx to a combination of FPGA and RT host VIs to provide the same funcionality. Data transfer from FPGA to host is made using two DMA FIFOs, one for vibration signals (modules NI 9233) and the other for voltage data (module NI 9205).

One problem arose on first development tests: when loading and running the FPGA VI sometimes DMAs seem to not start. I mean, sometimes application starts ok and host receives acquisition data, but sometimes "FIFO read" times out before transferring any sample. This happens randomly and once it starts ok, it remains ok.

We started using NI-RIO 2.4.0. Then upgraded to NI-RIO 2.4.1, but remains the same.

To workaround this problem, we introduced the next tricks:
- Before writing acquired data to FIFOs on FPGA, one known value is written to each FIFO at FPGA VI start.
- After launching the FPGA VI, the host tries to read the known "samples" from the FIFOs.
- If known "samples" are read ok, the program continues. If not, it aborts the FPGA and tries again.

Sometimes these two first "samples" are ok and data transfer continues ok, sometimes the first read gives two zeroes, sometimes gives one random value and the other read times out or sometimes both time out. On all "wrong" cases, data transfer times out.

This worked fine (while we wait for a new NI-RIO release), and acquisition begins ok in one to five attemps.

The problem we face now is that for our application we need to restart acquisition every 20 minutes, and when this is done for the third or fourth time, the CompactRIO crashes and reboots.

When rebooting it shows the message: "workQPanic: Kernel work queue overflow". We know this message is associated with incorrect interrupts handling, but we are not using interrupts in our app.

That's the first part. The ugliest part follows:

To make a better use of FPGA resources, we separated the channel calibration reading and placed it on a separate FPGA VI. This "calibration read" is loaded and executed once at the program start. Later, only the "data acquisition" FPGA VI is run.
The problem is that after loading, running and aborting the "calibration read" VI, when loading and running the "data acquisition" VI, DMA FIFOs definetively do not start, even with our workaround. After several attemps CompactRIO crashes and gives an "Exception Log". See attached console examples.

This definetively seems as a bug to us.

Any help would be greatly appreciated. As usual to LabVIEWers, we have a delivery date compromised so it is urgent to solve this problem.

Thank you,

Daniel R.
CLD
Chile

0 Kudos
Message 1 of 17
(5,102 Views)

Hi Daniel,

Any chance you can provide code that can reproduce this issue?  What controllers and backplanes are you currently using?  How are you passing your FPGA references around the application?

Regards,

Bassethound

0 Kudos
Message 2 of 17
(5,063 Views)
Hi Basset, thank you for your post.

We are using a 9014 controller, 9104 chassis, four 9233 IEPE modules, one 9205 module for voltage signals and one 9411 module for encoder signals.

FPGA reference is passed sequentially by wire. No global variables, no local variables.

We cannot provide the whole code of our project, but I will prepare a subset project with the code directly related to this issue.

However, we already extracted the FPGA-related VIs to a simpler program and ¡surprise!, they work fine, so we think this issue must be related to resources loading.

Thank you,

Daniel
0 Kudos
Message 3 of 17
(5,054 Views)
Hi all,
Here's the small project with the involved VIs. As I mentioned before, this works fine, but in the big application it doesn't.

Any comments from National Instruments?

Regards,

Daniel R.
CLD
Chile


Message Edited by Daniel_Chile on 06-12-2008 11:26 AM
0 Kudos
Message 4 of 17
(5,034 Views)
Hi Daniel,
 
I work on the CompactRIO Software team in R&D and have been investigating your issue.  I looked at your test application and like you mentioned it was not able to reproduce the issue you described on my systems.  Of course its much easier to root cause and issue if we can reproduce it, however, I had some thoughts I'd like you to try to get more details at what might work around the isse on your end and give us more information to start investigating what may be the root cause.  For starters I wanted to try a few things.
 
First, once the DMA reads get into the bad state, I would be interested if the application can recover by executing the FIFO.stop followed by a FIFO.start.  You could do this in the application like the following.
 
Second, put in a wait of about 100 ms before and after the Start method in the Carga_FPGA.vi
 
Last,  lets remove the reset and download method from the Carga_FPGA.vi
 
Let me know if any of the tests workaround the behaviour your seeing.  We can also try several of them together at the same time.
 
Basset Hound
0 Kudos
Message 5 of 17
(5,010 Views)

Here is the image of the Stop/Start FIFO methods



Message Edited by Bassett Hound on 06-13-2008 08:11 AM
0 Kudos
Message 6 of 17
(5,008 Views)
Hi Basset, thanks for your suggestions.

Unfortunately, we still have the same problems.
1. We introduced the Stop and Start methods once failed, but FIFOs didn't recover.
2. We introduced the wait before, after and between DMA FIFOs start. This seemed to show a better behavior (less start attemps to success), but didn't solve the problem. Sometimes it just doesn't start.
3. We removed the reset and download methods, but we didn't appreciate any change.

Can we send you the whole project, so you can review it and reproduce this issues? Obviously, I can't post it here.

Thank you,

Daniel R.
CLD
Chile
0 Kudos
Message 7 of 17
(4,962 Views)

Yes,  please contact our Technical Support and I'll make sure to sync up with them so we can continue debugging this issue.  If you can post your Service Request number that would make it easy for me to get in touch with the Application Engineer.

Thanks,

Basset Hound

0 Kudos
Message 8 of 17
(4,960 Views)
Dear Basset,
Yesterday I sent the project to Technical Support. I'm waiting for an answer and/or SR number.

As soon as I receive it I will post it here. Thanks a lot.

Daniel R.
CLD
Chile
0 Kudos
Message 9 of 17
(4,938 Views)
Sounds good.  I'll look for you post.
0 Kudos
Message 10 of 17
(4,934 Views)