Take the 2D FFT of a 278 MB array stored in a file? (Out of memory errors)

[Yevoc6942]

Hi all,

I have a 14416x19340 boolean array in a file that I need to take the 2D FFT of.  Since LabVIEW doesn't take FFTs of boolean arrays, taking the file and storing it in a U8 representation in memory will have to do.  Even then, making a few copies of an array that size should be no problem, right?

The only thing I've managed to achieve is initialize an array of the proper size and do no more.  Once I use the in-line replacement structure to replace the array values with the values in my boolean array file, the VI responds with "out of memory," which completely boggles my mind.

This certainly should be possible without even having to delve into memory management.  Can someone point me in the right direction on how to accomplish this before I turn to the laborious task of coding the whole thing in a language that gives me allocation control (and doesn't have FFT libraries)?  I have 3.2 GB of RAM to use, and the max I've seen LabVIEW at before it returns an error is using 680 MB of RAM.  (It normally has less than 500 MB before throwing the error, but it depends on which code I try)

I have restarted my computer in selective startup where nothing else is running and the computer is idling at 110 MB RAM usage before I start the program.  Even then, it gives a memory error with my in-line replacement code.  (In that code, I initialize a 2D array of the size stated above, then I enter a single for loop, make the array into a shift register, load a single bit from the file inside the loop, and then use the in-line replace structure where the pre-allocated array indecies are obtained via quotient and remainder of the loop iterator.  That code instantly returns a memory error.)

Thanks in advance.  I'm using Labview 2011 on XP SP3 with 3.2 GB of RAM.

[johnsold]

One thing comes to mind immediately.  Arrays in LV (and many other languages) need to be in contiguous memory spaces.  With arrays of that size you need to be sure that extra copies do not occur. The FFT VIs use DBL as the input representation and have complex outputs.  This means that you will have two arrays just for the FFT.  Since you are trying to read your file as U8, you have another copy. Now add it up. 278 MB of U8. 2.23 GB of DBL. 4.46 GB of complex.  

This will not work on a 32-bit OS.

You will probably need to break down the calculation into smaller chunks.  The FFT algorithm works that way so you may be able to find something which will work for.  What are you doing with the result of the FFT?

Lynn

[Pete.Dunc]

One quick addition - assuming a 32-bit OS.

Although you have 3.2 GB of RAM on your machine, by default LabVIEW can only access 2 GB of this.  If need be, you can extend this memory by following these instructions.

[Yevoc6942]

I haven't even gotten to the FFT issue. I'm just working on storing the array in a variable without crashing. I have to display the entire 2D FFT to a group in less than a week. Doing it in chunks is not an option.

[Yevoc6942]

I already did the /3 GB switch, and it's completely worthless. LabVIEW crashes before it even hits 1 GB. I figured memory fragmentation would be an issue, but when nothing else is running and the comp is using 100 MB just after startup before LabVIEW starts? Blaming fragmentation there is a bit of a stretch when the array is 278 MB. I can't even instantiate the 2D array by itself in a U8 representation without LabVIEW crashing and corrupting the VI when I try to save. It's not even worthy of being called a VI. It's just a single array control with zero code, and LabVIEW can't even do that. Once I make the U8 array control 14416x19340 in length, LabVIEW is already taking up over 800 MB just to hold that and crashes when I try to save? I'm sorry, but when did LabVIEW become this bloated? This is absolutely unacceptable in a programming platform. Let alone one this expensive. I understand now why programmers stick with text based. I'll be coding this in C.

[altenbach]

---

@Yevoc6942 wrote:
I already did the /3 GB switch, and it's completely worthless. LabVIEW crashes before it even hits 1 GB. I figured memory fragmentation would be an issue, but when nothing else is running and the comp is using 100 MB just after startup before LabVIEW starts? Blaming fragmentation there is a bit of a stretch when the array is 278 MB. I can't even instantiate the 2D array by itself in a U8 representation without LabVIEW crashing and corrupting the VI when I try to save. It's not even worthy of being called a VI. It's just a single array control with zero code, and LabVIEW can't even do that. Once I make the U8 array control 14416x19340 in length, LabVIEW is already taking up over 800 MB just to hold that and crashes when I try to save? I'm sorry, but when did LabVIEW become this bloated? This is absolutely unacceptable in a programming platform. Let alone one this expensive. I understand now why programmers stick with text based. I'll be coding this in C.

---

So seem to have some serious misconceptions. Your U8 array is a bout 280 MB initialized, but if you try to put that in an indicator, you also need a copy for the transfer buffer and the indicator, which explains the 800MB (remember, indicators retain their own copy, because they need to hold it while the code is doing something else with it). As soon as you wire it to the FFT function, it is coerced to DBL with the same number of elements, i.e. 8x larger, and a datacopy needs to be made. The FFT needs a certain working memory and the output will be complex (as Lynn already said), meaning you will have >24 times your 280MB, or about 6+GB just to do the FFT, not even counting the display to show the result. I would like to see you coding this in C on a 32bit platform. 😮

Can you show us your "simple" code?

It would really help to see some actual code. Run it with 10x smaller arrays, look at the memory usage, show buffer allocations, and do some profiling. Minimize the number of indicators and controls, don't use local variables or value property nodes, don't use probes on the big array, and disable debugging. Now modifiy the code for maximun inplaceness, minimizing the number of buffer allocations.

I no way will the full array work directly in LabVIEW 32 bit. Fortunately the 2D FFT can be done in slices (as Lynn already mentioned) so that's what you have to do.

I don't understand what you mean by ...

---

@Yevoc6942 wrote:
I can't even instantiate the 2D array by itself in a U8 representation without LabVIEW crashing and corrupting the VI when I try to save. It's not even worthy of being called a VI. It's just a single array control with zero code, and LabVIEW can't even do that. 

---

How did you "instantiate" the array? When saving the VI, the data is not saved with it unless it is contained in a diagram constant or as default value for the control. Is that what you are doing? When saving, the data needs to be compressed (you don't want a 300MB VI file!). How long did you wait? How did it crash? Was there an error message? 

[altenbach]

---

@Yevoc6942 wrote:
I'm just working on storing the array in a variable without crashing.

---

Please explain what you mean by "storing in a variable". This can mean many things in LabVIEW.

You seem to used text code based paradigms that may not be appropriate for a dataflow language. It is sufficient if the large array lives in a wire, anything else just creates copies.

[altenbach]

---

@Yevoc6942 wrote:
I have to display the entire 2D FFT to a group in less than a week.

---

I am not aware of any presentation display that supports a resolution of 14416x19340 pixels. What exactly do you have in mind?

---

@Yevoc6942 wrote:
Doing it in chunks is not an option.

---

Who specified this odd requirement? Unless you are operating on an FPGA, nothing happens all at once. Are you under the impressions that the 2D FFT primitive magically does not do it in chunks internally. A 2D FFT is simply a two step sequence of transforming all rows followed by all columns (or vice versa). It even says so in the help. This can easily be done in sequential steps with a much smaller working memory footprint.

• What exactly is in your boolean data?
• What kind of information are you trying to get out of it?
• Can you give us some background information on the problem?
• Why FFT? Do you just want the magnitude or is the phase important too? Maybe you only need a DCT?
• Are you interested in all frequencies or only a small subset of frequencies?

Depending on what you want, there might be simple algorithms that can operate on the U8 data directly.

How are you limited in terms of execution time? No matter what, transforming this thing will take minutes.

What exactly prevents you from using LabVIEW 64 bit?