"Not enough memory to complete this operation" & "Memory is full"problems with Labview 7.1

[gigi85]

Hello,

I'm a fresh engineer and i'm learning labview for my new job.

I have this problem...I've made a VI that open and elaborate a txt file of about 25K pages...

well my program works for normal file, but not for this huge one.

Labview send me many warnings during the execution of the program: "Not enough memory to complete this operation".

I press ok and after a little it tell me memory is full.

I use my data as a string and sometimes as an array of string, passing it by shift register in a event case structure.

If you have any ideas to solve my problem are welcome 🙂

__________________________________________________________________________________________________________________________

PS: I use Labview 7.1 and i can't send you my VI (it's property of the company where I work)

Thanks

[GerdW]

Hi gigi,

"pages" is not very descriptive for text files, better tell the actual size of the file...

- There's an AppNote on "handling large datasets" available. It should be included in the LabVIEW help too (I think even LabVIEW7 had that point in the help)!

- General answer on your generic question: read (and process) the file in smaller chunks...

[Sam_Sharp]

Are you reading the entire file at once in your application? You could try reading the file in chunks and processing the chunks individually - thereby limiting the amount of string data you have in memory at any time.

You can also do some optimising on how you handle/process the data. The key here is to limit the number of times you create duplicates of the data (e.g. by passing into a SubVI) - you could use a Data Value Reference to pass data around by reference (which doesn't create a copy in memory).

If you're working with large arrays - there are some array functions which are inefficient in terms of memory allocations.

There's a lot of information available here: https://www.ni.com/docs/en-US/bundle/labview/page/vi-memory-usage.html

(Particularly the part about showing buffer allocations which can help you see where your data is being duplicated in memory)

[gigi85]

the file size is 85mb 🙂

I have just read "Managing Large >Data Sets" by NI, and it invite to use a shift register global or a single element queue.

The document make some examples, but I don't have that document's VIs.

and I have never used a global var before O_o

as i told i'm new in using labview... 🙂 if you could suggest me in what way use and work with smaller chunks i'll appreciate 😄

Thanks again

[gigi85]

Are you reading the entire file at once in your application? Yeah 🙂

So i'll try to divide it 🙂 OK

Where can i find data value reference? i've tried to search it in labview help but i can't find it

anyway thanks for your link..i'm going to read it 😄

[Sam_Sharp]

For dividing the file - just read in a fixed amount at a time (you can set the number of characters to read) - depending on your data (e.g. the format - new lines at end of rows?) you will probably need to do some checking so that you correctly convert the chunks data from the file into 'records' (if you just read the first 10,000 characters - you might cut up some data and only get a partial record), for example by reading a certain amount, and then continue reading until you see the next newline character.

If you're text file is in a line-by-line format, you could always try this (although I don't know how that VI handles the file pointer - will a second read return the same results from the beginning of the file or will it return the *next* 200 lines etc.):
https://forums.ni.com/t5/Example-Code/Parse-Text-File-to-Line-by-Line-String-Array/ta-p/3497705

As for Data Value References - there's an example here: http://www.ni.com/white-paper/9386/en/ 

Of course - a lot of this is all dependent on the files you're reading and how you are processing it! Without an example / more information it's difficult to assist!

[Hooovahh]

Note that you won't find Data Value References in LabVIEW 7.1.  I think they came out in 2009 era plus or minus a version or 2.

[Sam_Sharp]

Oh drat! My fault for not noticing Gigi had said LV 7.1. My apologies!

[gigi85]

eheh no problem 🙂

one more question: in VI Memory Usage document say to avoid build array and concatenate string functions...

but they don't tell what i can use instead of them 🙂 do you know?

[Sam_Sharp]

Instead of Build Array, you should pre-initialise your array instead with the required number of elements.

I'm not sure about the alternative to concatenating strings...other than just to try and avoid building really long strings with it.

When you concatenate strings or use build array - LabVIEW has to allocate more memory to accommodate the new array/string size - which is inefficient for memory. Pre-initialising the array means that allocation only has to happen once instead of every time you use the 'build array' function.