Out of Memory Error, Buffer Sizes, and Loop Rates on RT and FPGA

[RyanTheEngineer]

I'm attempting to use an FPGA to collect data and then transfer that data to a RT system via a FIFO DMA, and then stream it from my RT systeam over to my host PC. I have the basics working, but I'm having two problems...

The first is more of a nuisance. I keep receiving an Out of Memory error. This is a more recent development, but unfortunately I haven't been able to track down what is causing it and it seems to be independent of my FIFO sizes. While not my main concern, if someone was able to figure out why I would really appreciate it.

Second, I'm struggling with overflows. My FPGA is running on a 100 MHz clock and my RT system simply cannot seem to keep up. I'm really only looking at recording 4 seconds of data, but it seems that no matter what I do I can't escape the problem without making my FIFO size huge and running out of memory (this was before I always got the Out of Memory error). Is there some kind of tip or trick I'm missing? I know I can set my FPGA to a slower clock but the clock speed is an important aspect of my application.

I've attached a simplified version of my code that contains both problems. Thanks in advance for taking a look and helping me out, I appreciate any feedback!

[speleato]

Hey Ryan,

A few general questions for you.

- What node specifically is returning an out of memory error?

- How much RAM do you have in your system?

- How much data is 4 seconds worth?

I'll take a look at your code and see if I have any more specific questions or suggestions.

Sebastian

[RyanTheEngineer]

Hello Sebastian! Thank you for taking an interest in my problem!

- I guess my inexperience with LabVIEW is showing here, but I wasn't aware that the error gave any information on where it was occuring. I just ran my code again and received the "Not enough memory to complete this operation" it doesn't seem to give any indication on where or why it occured. Maybe I'm just missing something?

- I have 8 GB on my host PC and 4 GB of RAM installed on the RT target of which only 3.1 GB are available (I despise 32 bit systems, otherwise I could just throw some more RAM at this problem)

- I am sampling at the same speed as my FPGA clock (100Mhz) and have 2 channels. Since I am using the I16 data type, every sample should be represented by 16 bits and there are 8 bits to a byte, the total size should be given by the following formula...

     (100,000,000 * 2 * 4 * 16) / 8

Which yields 1.6 GB. Ideally, I could just make my buffer this big and not care. Unfortunately though, I will eventually need this RT target to do other operations as well so I need to try and preserve memory space.

Please just let me know if there is anything else that would assist you! Thanks again for taking the time to help!

[TYtanic]

Hey Ryan,

Can you also give us a little information on what hardware you're using? From the FPGA VI, it looks like you're using a Single Cycle Timed Loop with analog input, but that shouldn't be possible with analog so it'd be nice if we had some info on what RIO and Modules and LabVIEW versions you're using.

Also, does the FPGA immediately overflow?

[RyanTheEngineer]

Yes I would be happy to give you some of hardware and software details...

In a PXIe-1082 Chassis I have...

- 8133 Controller (RT target)
- 7965R FlexRIO FPGA with a 5781 Adapter Module

I am running LabVIEW 2014 SP1 on my host PC

Please let me know if there was any other details you needed!

And yes, I am reading from an analog input on my FPGA in a single cycle timeloop. In a prior version of this code we had successfully read a sinewave in from a function generator, so I feel confident that aspect of it is working correctly. Or, I should say I felt confident until I read your comment. Why should this not be possible?

[David-A]

The 7965 can't stream to a host controller faster than 800MB/s. If you need 1.6GB/s of streaming you'll need to get a 797x FPGA module. 

What FAM are you using? You may be able to do some bit packing and reduce the required throughput if you've got a 5772 for example which only has 12 bits of precision but pads an I16 with a few bits. 

The 7965 has 512MB of DRAM. If you're generating 1.6GB/s of data you can store a little less than a half second worth of data. You may be able to get up to a full second if you buffer through BRAM, through the DRAM, and then read out of the DRAM as you're writing to it and stream to the host controller.

Once you do get the data to the RT host, you'll never be able to stream 1.6GB/s of data over the ethernetport. The enet port on the 8133 is a 1Gige port, which means a maxmium theoretical of 125MB/s of data, and an achieveable rate thats likely much lower than that. So once it gets to the data reaches the RT Host you'll need to store it in memory or write it to disk and then transfer it over the enet port. 

Also FlexRIO acquires data from its IO differently than cRIO or R series does. So you're doing it correctly by placing the IO node inside of a SCTL that uses the data clock that comes from the CLIP. 

[speleato]

---

@David-A wrote:

The 7965 can't stream to a host controller faster than 800MB/s. If you need 1.6GB/s of streaming you'll need to get a 797x FPGA module.  

---

I believe the application calls for 1.6 GB over 4s, so 400 MB/s, which should be within the capabilities of the 7965.

I was going to say something similar about streaming over ethernet. I agree that it's going to be necessary to find a way to buffer most if not all of the data between the RT Target and the FPGA Target. There may be some benefit to starting to send data over the network, but the buffer on the host is still going to need to be quite large. Making use of the BRAMS and RAM on the 7965 is an interesting possibility.

As a more out there idea, what about replacing the disk in your 8133 with an SSD? I'm not entirely sure what kind of SATA connection is in the 8133, and you'd have to experiment to be sure, but I think 400 MB/s or close to that should be quite possible. You could write all the data to disk and then send it over the network from disk once the aquisiton is complete. 

Sebastian

[speleato]

Or maybe something like https://www.ni.com/en-us/shop/category/pxi-platform-completion-hardware.html?productId=118764

[RyanTheEngineer]

Wow, it did not occur to me that my ethernet connection would be so slow and streaming over network would be my limiting factor. Bad assumption on my part and thank you guys for pointing that out. I will start investigating the use of SSD drives for our controller or perhaps something like the 8260 Sebastian linked me to.

For now though, I guess I will make do with obnoxiously huge buffers on my RT target so I can at least collect some data. My first problem still remains though, why is LabVIEW telling me that I am running out of memory, even when I'm using small buffer sizes?

[speleato]

On the topic of the out of memory error, the first step is to figure out where it's coming from. One of the VIs in your diagram is setting that error on the error wire and it's getting propagated to the simple error handler.

I'd suggest using probes or indicators to figure out exactly where and when it's coming from and we can go from there.

Sebastian