LabVIEW
cRio - rt shared variable multi element fifo data loss
Hi,
Ive looked extensively through the forum and labview resources / google but am still confused by this problem.
I am acquiring data on FPGA and RT. On the RT side, I am acquiring / logging 45 channels on various modules through a shared variable with multi element FIFO enabled in the typical fashion. Regardless of the relative loop times or the number of arrays in the multi element FIFO (have tried 2 to 1000), I always seem to get random data loss. I include elapsed time and loop time in the logged array, and looking through the data, whilst the loop time is always correct, the elapsed time jumps intermittently by twice the SCTL period. I had timeout set to 0ms and adjusted the logging loop time such that I did not see any indication of timeout.
I then tried a single element RT FIFO out of curiosity and the skipping apparently stopped.
The method for calculation of these times I am using seems pretty standard [tick count in the loop - tick count before entering the loop for elapsed time and tick count & shift register for loop period], so i am curious why in the data, the loop time never indicates any skipping, which is only apparent from the elapsed time.
So I have various questions which are probably obvious to seasoned labview veterans but not to me...
1) Why does the 'data loss' only seem to happen with the multi-element FIFO?
2) How do I determine an appropriate number of arrays for the multi element FIFO? I wasn't able to find any suggestions in the documentation.
3) Why do I even need a multi element FIFO to log data? (I used it because it seemed sensible to have a buffer and most examples use it)
4) I have often read people suggesting that the logging loop should run as fast as possible to 'suck' data from the buffer, but when I do this I just get lots of repeated values and still have data loss. I found that best results (in my case) were with the logging loop running slightly faster than the acquisition loop.
Sorry but I can't post the full code (IP). Here are snippets.
Thanks in advance.
Phil.
Is the second snippet the Spaceplane Demonstrator/convert-rt-modules.vi?
I guess maybe not, given the loop structures, but if not then it's a little unclear what the first snippet tells us.
You also mention your FPGA code and indicate your skipping may be related to the SCTL period, can you show any of that code? Or you don't believe that's relevant?
Adjusting loops to make them run at similar rates without actually handling a change in their rates probably isn't very robust - the general advice regarding having your consumer running as fast as possible is good provided you check if the element is new. Use of a RT-FIFO can enforce this (if timeout -> empty -> don't log), or indeed a general queue (same basic idea). Having it run without any loop-limiting step will use a full CPU on RT though, so take care to avoid that (i.e. set a timeout value != 0, and take care with the option at FIFO Create (polling with !=0 timeout still uses full CPU, blocking does not - if you want polling then set 0 timeout and a wait in the timeout case perhaps)).
To give a better answer, can you tell me what's happening in the subVI in the first snippet (assuming it isn't the second snippet).
Where is the second snippet from?
How is the FPGA involved (because your scan rate loop suggests scan mode acquisition for the RT 45 channels).
When you talk about multiple channels and a multi-element FIFO, are you using the FIFO with 1 element per channel, or multiple elements for one channel? If the latter, do you have multiple FIFOs? If not, how are you reshaping the array to ensure the correct channels/samples at the receiver/logger?
Hi, thanks for your reply.
So the snippet which contains the SCTL is reading data from modules, and either feeding directly into a 'raw data' buffer, or applying some ranges to the raw data (4-20ma to pressure) and feeding into a 'calibrated data' buffer. Only the raw data is logged. The broken wire feeding into the SCTL is just an array of gradients and intercepts used for converting raw data to pressures. Here is the contents of the subvi in the RT SCTL;
My array of raw data is fed to a shared variable: array of double with multi element RT FIFO. 10 arrays, 45 elements. As I said before, I am still unsure what is appropriate for the number of arrays. The output of the buffer is reshaped just before the TDMS write, as shown in the snippet containing the while loop of the original post.
The FPGA side of things is functioning normally and I don't believe it has an influence on this RT logging issue.
I've now implemented the condition for logging with timeout = false.
Hope that provides some clarification.
Thanks.
Phil.
Hi again, thanks for the increased detail.
As a quick note, a SCTL (Single-Cycle Timed Loop, which you probably already knew, but for others reading) is exclusive to the FPGA code - on Windows/Linux/Mac or Linux-RT, it's just a Timed Loop.
A Timed Loop gives you information about the timing and aims to follow a specific schedule (on RT it's usually pretty good at that unless the code inside will take longer than the loop period) but it won't break if it fails (you can set various options to control the behaviour in this case, see here: Timed Loop Configuration and here: Setting the mode of a Timed Loop)
On FPGA, if a SCTL (there are no normal Timed Loops on FPGA, just Loops or SCTLs) cannot be programmed to execute in a single cycle (of whatever clock you set) then the compilation will fail with a Timing issue. So in this case, it will always (assuming a successful compilation) run within the allocated time (25ns for the (I think) standard 40MHz clock).
So in your Timed Loop on RT, you could check things like the Finished Late input if you think the timing is going wrong. It also gives information about the time taken etc (this should be less than the period when not late IIRC, it's the required time, not the spent time) but for the previous iteration (which makes logging a little more tedious).
I think based on that idea, you might be able to get an answer to your first question (before the numbered bits) - "i am curious why in the data, the loop time never indicates any skipping, which is only apparent from the elapsed time". If the loop takes too long, you may have it set to skip missed iterations but retain original setting. This will make the loop time fairly close to constant (assuming only a narrow miss) but could change the total time (since the loop start). The second link above makes this more clear than my explanation...
Regarding your numbered questions, I'll take a shot:
- Not sure. Maybe some behavior of the timed loop combined with extra delay from the multi-element (where a single element to write is an array of 45 doubles, not the same terminology as in the dialogs, sorry) SV writing? 450 doubles probably isn't enough to cause issues with memory or similar (unless you're very very full on the RT system).
- Typically I'd suggest choosing a buffer length based on an expected time. So if you have good reliable network and aren't worried about storing buffered data, less than a second should be fine. Otherwise, maybe you'd like a couple of seconds to reconnect etc. Then, convert your chosen time to a number of iterations at your loop period (scan rate). You can do this in the Network rather than FIFO settings if you want though.
- If I understand correctly (might not), you don't but it allows you to write repeatedly into the FIFO without blocking, even if the reading end has slowed down a bit. If you use a fast reader (and can keep up with acquisition) the multiple arrays are probably unnecessary.
- I think we already covered this, and you mentioned that you'd handled the timeout=false logging vs timeout=true. Does that fix this issue/answer this question?
