Efficient data saving in RT target

[Michele-Icing]

Dear community,

I would like to ask a question concerning efficient (as fast as possible) data saving of sensors data in the RT target. I know that the following issue has been discussed in previous threads, but I was not able to apply existing solution to my case study. Therefore I would like to ask further clarification.

The case study is as follows. We utilise a CRio 9114 chassis with NI 9220 module to measure AI from either accelerometers and strain gauges. Therefore my starting point was the NI 9220 getting started example. Following the snippet of the FPGA. This is basically as in the example project unchanged. 

Following, we send the data to the RT target (snippet below)

 This is as well very similar to the example. I just use this VI to sort the data in FIFO of fixed length (455 elements in this particular example). The array in the FIFO is then decimated and sent  to the global variable. Until here all good. The VI runs at the desired sampling frequency and I do not experience timeout. 

I have then built a "slave" VI specifically to save the data. In my application, I have to save the data in the RT target. The VI I have written for the purpose is the following

On the left I prepare the header for the TDMS file, then I read the global variable, average the data for each channel in order to obtain a single value out of the incoming 455 points of the array, save the first TDMS group and then add a second group for the date and time at which data are acquired.

The problem is here. I cannot acquire faster then 50 Hz. I know that the post-processing (average) will take some resources, but this is definetely too slow. The data are in fact clearly being visualised at a much faster rate, but the array is not saved accordingly.

Can you point out how to make the data acquisition faster (my aim is to save at 1 kHz rate if possible) or at least more efficient? Am I using some functions in the wrong or inefficient manner?Is there any mistake in the VI structure? Is it ok to use a global variable to pass data between VIs in the RT level?

I have also read in the previous threads that the "NI_MinimumBufferSize" property should help, but it is unclear to me how to use this in practice (in my example this is definetly wrong).

Any advice is highly appreciated and if any extra information is needed, I would be happy to clarify further my problem

 

[ZYOng]

Local or local variables do not buffer data. You will encounter data loss. Instead, you should use a queue to pass data between the acquisition loop and logging data.

I would suggest you to use Embedded High Speed Data Logger Reference Design

[Michele-Icing]

Hello ZY Ong,

sorry but I do not think that this answers my question. If you break down the functions used in this example, it all comes down to standard RT functions, namely the TDMS library and also queues are used.

I agree that the RT queue can help, but I tried implementing it in a previous version of the code I shared in the post and there was no improvement compared to the global variable (at least when it comes to how fast I can save the data). 

The interesting thing I notice in the RT data logger is that a state machine is utilised to initiate, record and close the saved file.

Is the state machine going to noticeably increase the efficiency in data logging in the RT target?

[ZYOng]

Sorry I didn't have time yesterday to provide a detailed explanation of your question.

The latency and performance have nothing to do with the use of a state machine but are largely affected by the loop rate. We need to first understand how data is sent to the FIFO, read from the FIFO and write to disk.

Assuming that you want to read NI-9220 at the maximum rate for 8 channels. You would be generating 800kS of data per second. Remember that it is always more efficient to send a big chunk of data. So if you want to write data to TDMS every 100ms, you should read 80kS of data per FIFO read. And your host FIFO size should be at least twice the size or 160kS.

So basically, your current FIFO reading rate is too slow to keep up with the data generated by FPGA. RT Queue does not help but it is the correct way to do high-speed streaming. Variables should never be used in  streaming because they are not buffered.

For more details, see 

How DMA Transfers Work (FPGA Module)

Difference Between Configuring the Number of Elements in a FIFO and Requesting the Depth of the FIFO

Stream high-speed data between FPGA and PC with a DMA FIFO

Embedded High Speed Data Logger Reference Design encapsulates all of these nicely and you don't have to spend time and effort to create everything from scratch.

[Michele-Icing]

Hello,

thanks for the kind and detailed reply.

I have implemented a queue using a consumer producer architecture and the queue itself seems to work fine. Still I cannot save at the desired rate. I believe the problem is, as you mention the fragmentation once I use the TDMS functions. I am trying to use the NI_MinimumBufferSize property, but I still do not get it right. I am probably calculating wrong the size of the buffer.

I will need a couple of days to study the project you sent in details. In the meantime, if you know of a way to calculate programmatically the buffer size based on the FIFO DMA size that would probably solve the issue.

Thanks again,