Is there a quick and easy way to reorder de-interleaved data?

[rjmartinator]

I am collecting 5 channels of data through a DMA FIFO by interleaving, then recording to a TDMS file. When I check it, it's in mostly good order, but there are certain points where the channels get switched. For example, the Channel order when I open the file is 1 2 3 4 5, as expected, for several thousand elements. But at some point, it gets shifted to 4 5 1 2 3, and later 2 3 4 5 1. Left-to-right, they are always ordered correctly, so it's easy to reorder them, but very tedious with millions of elements. I have attached some sample data. I'm looking for A) a way to prevent this or B) some kind of sorting tool or script in LabVIEW or DIAdem that may sort it for me. I'll work on a sorting script in the meantime. 

I am using LabVIEW 2011 Real-time and FPGA 2011, and the standard TDMS open and write functions. I also have access to DIAdem. 

[xseadog]

Hi,

my initial thought is that you are losing data. Possible due to buffer problem, could be timing

Suggest you try a free running counter on the fpga configured with 4 fixed constant values in the array ( total 5 in all)  so there is  initial only 1 value changing. See what effect that has !

Regards

xseadog

[rjmartinator]

It seems that that is part or all of it. It was set up with a free counter to help organize the data. The interval is (should be) always 16 or 17 ns between samples. Sometimes this is still correct at the shift points, but other times it is off, with a maximum interval of 33 ns in what I've looked at so far (double normal, or one missed sample). I don't know if this still indicates data loss due to buffer size, I thought more than one sample would be lost if this were the case. If it is, I'm not very concerned about losing the one sample. Should I just develop a sorting algorithm then?

[ColdenR]

If you don't care about losing individual samples, then yes a sorting algorithm that recognizes the differences between the five channels could work.

However, you should be able to avoid losing data coming out of your DMA FIFO. We would need to identify the place the data is being lost. How is your code set up? Are you reading from the FIFO and writing to the TDMS file in the same loop? Or, if there is an issue on the FPGA side or instrument side, how are you reading the data into the computer or RIO in the first place?

Especially if you're using an R-series card and are doing the TDMS file write on your computer, then you should make sure that you use a producer-consumer architecture (check out http://zone.ni.com/devzone/cda/tut/p/id/3023).

[rjmartinator]

Sorry for the slow response - 

I've looked at everything, and so far I can't identify the problem. I believe my code is consistent with the producer-consumer architecture. 

On the real-time side, I am reading data from the FIFO and writing to the TDMS in the same loop.

On the FPGA side, analog channels are read, converted to the same fixed-point format with a time stamp in a single-cycle timed loop, and then ordered into the FIFO in a for loop. I have posted the relevant parts below.

I am using C-series modules in a CompactRIO.

[srm@viewpointusa.com]

if you check the status of the timeout of the FIFO write, you will see if your FIFO is overflowing.  you can increase the FIFO size on the FPGA and increase the queue size on the RT to make it more tolerant of potential delays.  also your FPGA loop should use feedback nodes after the read from i/o node to pipeline the processing of the data and make the loop timing more consistant.  also put 0 timeout on fifo write.

[nathand · Accepted solution]

Looks to me like the problem is that you do not remove elements from the FIFO in multiples of your channel count.  The Decimate Array function always returns equal-length arrays.  If the size of the input array is not a multiple of the number of output arrays, it drops elements, and that's where you're losing data.  You should round down the number of elements to read to the nearest multiple of the number of channels - I would insert a "Quotient and Remainder" function with the divisor as the number of channels and subtract the remainder from the number of elements.

[srm@viewpointusa.com]

nathand is correct.  i missed that.

[rjmartinator]

I'll implement that then, thank you!