how to optimize the memory for a large labview array?

[8139741635]

I am running LabView on Windows XP with 4 Gb of RAM. The array is 51X256X256 complex numbers, but I want 300x256x256. The program runs out of memory when I use more than 51-53 arrays of 256X256. I do not know how to use the in place element structure to modify the vi below. The unbundled/ bundle structures do not present symmetry. Anyone can help me with this? 

Thanks!

Mariana

[Norbert_B]

Mariana,

is the VI, which contains the code you posted as screenshot, the only one in memory during execution? If so, can you please post the VI?

If the VI contains only this code and is the only one in memory during execution, the issue is rather odd. The array should take something about 26MB. The array is copied three times in your code (if i counted correctly), so you should have round about 100MB for this VI. So LV shouldn't run out of memory.

In order to reduce memory needed:

- Do not display large datasets (cluster and array) since they will create a copy (true if frontpanel visible).

- Use flat datastructures. So working with large arrays can require the arrays to be not packed in any clusters.

- Use functions which can reuse dataspace instead of functions which have to reallocate.

- Get used to preallocate any dataspace needed and reuse them. 

In regard to your 4GB RAM:

A 32bit OS cannot handle more than 2GB per application by default. LV 8.5 introduced "Large Memory Awareness"  to LV. It is possible to extend the memory used by a single application to 3GB using an entry in the boot.ini. This option is called Large Memory. Applications capable of using this setting have the "awareness". But with 32bit OS, it is not possible to allocate more than 3GB even using this setting. And please note that this might be a temporary working solution, but if the arrays get larger, you will run into the same issue again.

hope this helps,

Norbert 

[8139741635]

Norbert,

 

The full data3Dc array takes about 15.5 MB and the data3Dc comp.vi takes 31 MB. The empty data3Dc comp.vi array takes only 18.2 kB. This array is a variable that modifies at each scanned wavelength. I have a scanning wavelengths software package. For one wavelength I record a hologram followed by the reconstruction of the object field (1x256x256). 3D optical fields are added all together to create a big 3D optical field (Nx256x256). The code data3Dc comp_1.vi is not the only one in the memory. This VI is a subvi for other codes. A few other examples are FFT, IFFT, filtering, phase matching, cross-section, calibration….I did not care about memory optimization until now. So, I want to scan more than 50 wavelengths. By now (N=50) I have to restart the LabView after each scanning, do the reconstruction of the fields, start the LabView again to see the results. I think the problem is caused by this VI that is a subvi for other codes. I already extended the memory to 3GB in boot.ini, but the thing is that I want also a small scanning time. It doesn’t work. Extending the memory to 3GB makes the OS slower than before. My question is how to modify this array (data3Dc) or the VI (data3Dc comp.vi) to access lesser memory. I have tried to attach the full VI but it tells me that it is too big. I attached the empty one.

 

Thanks,

Mariana

[Norbert_B]

Mariana,

we got a little problem here:

You post a VI which can be improved, but one iteration of this VI does not waste all the memory. So the problem definetly starts somewhere else.

Since the VI is non reentrant, this is not the reason as well.

Please use VI Profiler (Tools >> Profile >> Performance and Memory) to seek out the VI wasting the memory.

hope this helps,

Norbert 

[altenbach]

There are a couple of things you are not telling us. How are you building the 2D array from the 1x256x256 slices? For this is is most optimal to allocate the full 3D array (Nx256x256) once and then replace slices. If you built the array one slice at a time, you need to constantly reallocate, which is very expensive and you'll run out of contiguous memory.

[DFGray]

You can get a lot of tips for handling this type of memory problem in the tutorial Managing Large Data Sets in LabVIEW. You need to be very careful with the Nx512x512 array after it is fully created.  Note that your full array size (300x512x512 complex numbers) takes about 1.2GBytes or RAM (300planes X 512x512 X 16bytes/point).  There is not normally that much contiguous RAM available.  To reach your goal, you will probably need to store your data in chunks.  I would recommend storing it in 512x512 planes using the single-element queue method or the reentrant LV2 global method mentioned in the tutorial.  Your top-level data object will then be an array with N elements containing either queue references or VI references, depending upon which method you choose.  This will also help you utilize fragmented memory in a more efficient manner.

Read the tutorial.  Step through your code with the OS memory monitor on to find out where your real problem is.  Let us know when you have further questions.  Good luck!

[F._Schubert]

Mariana,

I would look for a memory leakage due to some unitialized shift registers somewhere. They might fill up and keep the data each z-Cycle (propably even y-Cycle) of your scan. The vi you posted will deallocate the memory if not used any more. As you're able to do analysis of your data after restarting LabView, such uninitalized shift registers will be empty then.

Felix

[8139741635]

Hello All,

 Thank you for your valuable comments. The think is that I am so bad with this memory optimization. The array3Dc_comp is in the second part of my application (the reconstruction process). I started from the very beginning with the recording of the holograms part. So, I want to build up a 3Darray of 2D holograms. The 3D array contains all N 2D holograms and the increments dx, dy, dz. The question is: if I initialize the 3D array of holograms only (HH), do I have to initialize the 3D array that contains all holograms and the increments dx, dy, and dz (HHH 3D)?  I used the VI Profiler (Tools >> Profile >> Performance and Memory) to seek out the VI wasting the memory. The problem is with the arrays. I started from the beginning to allocate memory for the whole 3D array of holograms only (HH) and to use queue/dequeue functions. Something is wrong because I got an error at the end of scanning. What is wrong? I am using for the first time these functions. Please, see the attachment.  

Thank you,

 

Mariana

 

[altenbach]

Most of your code is extremely wasteful and it does not surprise me at all that you run our of memory.

Why don't you attach the VI instead of images? It is impossible to evaluate the code from a few images, because 90% of the code is still hidden in other sequence frames.

A few things I noticed:

• In the original version, your are building the 3D array at a while loop boundary, this is very expensive because LabVIEW cannot know the final dimension.
• Later you are constantly changing the data structures of the huge array, requiring more datacopies of the 3D array. (You go from U8 to SGL, to part of a cluster)
• What's up with all these local variables. (For example in the first image, why do you read Nxy from a local. The terminal is right there. It is not illegal to wire across structure boundaries! Similar problems everywhere.

In the last image, you are creating an array of queues. Makes no sense! Don't autoindex the queue wire at the FOR loop output! No wonder things are invalid. You are not preallocating anything this way, except waste even more memory. Your method with queues and inplace structures is completely misguided. Get rid of all that rube goldberg code! 😮

To preallocate the full size array, initialize a shift register with the Initialized 3D array and use "replace array subset" to fill in the slices. Try it!

[8139741635]

Hi,

I have attached the vi's. The main vi is WS acquire HHH.vi. You need a Mike Controller and IMAQ camera. I created a shift register.

Thank you.