cRIO channels being "changed"

[jeff_scharpf]

Ok this is a weird one that I'm sure has an explanation. I'm probably doing SOMETHING wrong lol. (LV8, FPGA, cRIO)

It appears that sometimes the data from CH0 is "switched" with CH1.

I have a simple setup. I'm reading 2 analog channels from a 9215. I'm using a DMA FIFO.

I used an NI demo program to build the FPGA portion, which is just reading CH0 and CH1, then putting them into a FIFO.

I've changed the FIFO timeout to various settings (0, -1, etc..) and the problem is still here.

The data seems to be read correctly in the FPGA, and seems to enter the FIFO correctly. I showed this by just displaying an array inside the FIFO to make sure the interlaced data looks good (CH0, CH1, CH0, CH1, etc..)

Using the NI example in my Host, with the Host appearing to keep up with the FIFO ("backlog" number is 0 or low), the data coming out of the FIFO will have the two channels switched. So the data at CH1 shows up first.

In fact, when doing 4 channels, I found the array coming in to be CH3, CH0, CH1, CH2.

Now, I'm SURE I'm doing something wrong. The host Timeout is set to -1. I've tried various FIFO sizes, various read block sizes, and various timeouts.

The strange thing is that it doesn't always do it. I can't seem to nail down what I'm doing wrong.

Any ideas?

Jeff

[preston_johnson]

How big is your DMA FIFO?  It may be that your RT side is not getting the data from the FIFO memory buffer fast enough and that the FIFO is being overwritten.  This might make it appear that the DMA channels are being switched around, if one channel element is overwritten.  If this is the case, two options

1) make a bigger FIFO

2) check when writing into the DMA FIFO that the FIFO is not full, it it is, write the same value again. 

[jeff_scharpf]

The example provides a "backlog" display, which I believe shows the number of readings still in the FIFO, is always low or at zero, so I don't believe that it's getting full.

The FIFO on the FPGA is about 16000 and the FIFO on the RT is 32000.

I will try your suggestion.

Is there any benchmarking to show how performance is affected if you create two individual FIFOs (one for each channel) versus one FIFO for both?  I guess I would assume that it would slow it down.

Thanks,

Jeff

[jeff_scharpf]

Here are my two VIs. I've tried various values for the timeouts.

[Bassett_Hound]

Hi Jeff,

You might also check if the FIFO times out.  If the FIFO times out, the data won't be sent and you may experience the channels jumping like you described.  You can try two things if this occurs:

1.  Slow down the Loop rate in your FPGA VI or...

2.  Increase the FIFO timeout.

Have you had the chance to try the other suggestion yet? 

Regards,

S. Bassett

[jeff_scharpf]

Changing the timeout to -1 seems to have helped.

I did not try the suggestion as I'm confused how to check the Full indicator without writing to the FIFO.

I have increased the FIFO size and it also helps.  Not sure what the maximum size is.

I only worry that I will get intermittent "hiccups". I'm not sure if that's just something I have to live with or not.

So far it seems to be working much better with the -1 Timeout.

Jeff

[Dustin_W]

I've run into the behavior you're describing before as well.  For me it was that my DMA FIFO was timing out on the FPGA. 

Let me clarify what happens on the FPGA and host.  I'll use an FPGA writing 5 channels (ch 0-4) as an example.  Let's also say that the data being written for each channel is constant (a-e).  If the host is keeping up, everything works great.  The FPGA writes a, b, c, d, e, a, b, c, d, e, a, ... for channels 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0,...  The Host then reads a, b, c, d, e, a, b, c, d, e, a ...  and assigns the data to channels 0-4 correctly - Perfect!  The problem is if the Host isn't keeping up.  In this case, the FIFO begins to fill up, and once full, the FPGA will try to write the value up until it's timeout value, at which point it gives up.   Data will never get overwritten in the FIFO, rather the data point get's dropped.  Lets say in our case we timed out when writing ch. 3 on the first iteration.  The FPGA would write a, b, c, e, a, b, c, d, e, a  The host doesn't know that the FPGA timed out, so as it reads items out of the FIFO it reads a, b, c, e, a, b, c, d, e, a.  As far as the Host is concerned it read data for chan 3 (in this case 'e'), it just happens to be channel 4's data, it then reads chan 0's data - 'a' for chanel 4, and so on. The whole input stream gets shifted.  There are two ways to get around that:

1 - Use a timeout of '-1'.  This ensures that your data will never get dropped, the FPGA will wait forever to write an item to the FIFO.  This has the unpleasant side effect of apparently hanging your FPGA I/O loop with no feedback, but it is clean.

2 - Monitor the timeout boolean and rewrite the same value if a timeout occured.  This can be done by placing your FIFO inside a Single Cycle Timed Loop as shown in the attached .jpg.  If you use this method you might want to also somehow latch the full state so that you can report that the host is not keeping up - this is done it the NI 9205 Advanced IO shipping example.

As a side note, your host I/O loop not being fast enough is not the only reason you might get a full FIFO.  This can also happen if the FPGA begins writing to the DMA FIFO right away rather than waiting for a signal from the host to start.  Often there is enough time between opening and running your FPGA reference and doing your first DMA read to cause the FIFO to overun on the FPGA side.

Hope this helps!

Dustin

Message Edited by Dustin W on 03-20-2006 04:39 PM

[jeff_scharpf]

Excellent!
Thanks for the detailed explanation. I was thinking that it was something on the FPGA end, and now I see it.

Yes I changed to a timeout of -1 and that helped so I understand what you mean now.

Correct me if I'm wrong, but the only issue with this is that since the FPGA will wait forever, the actual data will  have "missing" points in it (data not captured while the FPGA was waiting) if the FPGA has to wait.

If this is true, I would think that there would be no benefit to watching the "Full" state, since in that case it's just writing the same point over and over until the FIFO is NOT full. (Is this correct?). So then I would think you've already missed some "live" data at the channel.

Either way, I understand now, bottom line is if the RT doesn't keep up, then then data will be missed no matter what. I am ok with this understanding. Since my application just triggers the FPGA VI to run when I hit START, I think it's fine the way it is.

I will leave my timeout at -1 for now. I think the only problem I have is in the very beginning. (When I initiate the FPGA to start running). The RT seems to not be able to catch that first set of data. After a second or so, it "catches up" and everything is fine.

Thanks again for all the help!

Jeff

[Dustin_W]

I'm glad to hear that helped!

The reason to monitor the "Full" state is so you know you potentially have missing points.  This is important if you are doing waveform analysis, where missing points could throw off your calculations.  In some cases it does not matter if you miss points now and then, so you can get away with just using a '-1' timeout.

Note that if you do choose to monitor the "Full" state from the host, you should latch it on the FPGA.  If your FIFO is only intermittently full, there is a good chance you would miss it.

Dustin

[Kelvador]

I am having the same type of problem where channels just seem to be dropped and I tried all of the suggestions in this thread with no luck.  I'm having a problem with my timed modules.  I'm reading a 9234, a 9239 and a 9229 and transferring the data to my RT host via Target to Host DMA.  I'm exporting the onboard clock of the 9234 to the other two modules and setting the data rates equal (2.048 kS/s).  In my RT host I'm determining the number of elements to read based on the number of elements remaining on the previous iteration (using a shift register).  My elements remaining are typically around 900 while the FIFO is set up for 16,383 elements so I'm not filling the FIFO.

Any suggestions on this one?

Thanks.