Minimize Vi memory usage with data buffers?

[mcg123]

I am writing a Vi to manipulate and concatenate multiple large text files and then write the larger array to a text file.  However, as I have started to input more files, I have run into memory usage issues.  Does anyone have any suggestions for minimizing memory usage of this Vi?  Perhaps a way to buffer the data until memory becomes available? A typical input file contains 17 columns and rougly 500,000 rows of data (~50-60MB)

Thank you in advance. 

[mcduff]

Tips for improving memory usage:

1.) Don't use read/write to spreadsheet. It makes data copies. Write your own from primitive functions.

2.) Watch you wire branches, you make copies there.

3.) Certain functions copy data - insert into array, delete from array

4.) Write your data to file in small chunks, this avoids large data copies.

It is hard to follow exactly what you are doing from your vi. The diagram could use some cleanup.

Cheers,

mcduff

[mikeporter]

First if the data is numeric with large datasets you should be saving it as numbers -- not as text.

Second, you can get some memory management hints here:

http://www.notatamelion.com/2015/07/31/plotting-large-datasets-without-waiting-forever/

Mike...