LabVIEW
Speicherauslastung ermitteln mit dem Werkzeug "Leistung und Speicher"
Is the 16MB a binary file? What are the dimension(s) of the resulting DBL array?
simulation wrote:Zusätzlich wird das Signal zwei Mal intergriert und jeweils auch in XY-Grafiken dargestellt.
Die Messdaten werden zuerst in ein Arrays eingelesen und danach weiterverechnet.
You seem to have a lot of copies of large arrays in memory (raw string, raw (DBL?), transformed(complex?), x(DBL), as well as the front panel data structures to display 16MB of data in xy graphs and maybe arrays and probably some extra copies for wire branches, depending on the programming style.
Can you show us some code? Why do you need xy graphs? Typical data is equally spaced in time and this the FFT is equally spaced in frequency. Most likely you can use waveform graphs with proper x0 and dx setting. No front panel can show that much data at once, maybe you should decimate for the display?
Also remember that local variables (and such) cause extra datacopies in memory.
Arrays need to be contiguous in memory, so you might also run out of contiguous space well before you run out of memory.
What is your OS (Win7, Vista, XP, 32 or 64bit, etc.)?
- last edited on by
Content Cleaner
@simulation wrote:
The CSV-File is not binary, it's a ASCII file with two rows (time and value) and 770.000 Colums (16MB).
OK, assuming DBL, this is 770000(columns)x2(rows)x8*bytes/value) ~12.3MB memory footprint per data copy. Does this sound right?
For these many points, you should really use binary. Parsing a text file is significantly more expensive.
If your computer uses 1GB of memory for this, you must have about a hundred data copies in memory.
Lets' count: 1 copy as string during reading, one copy as DBL input, one copy for x, two copies for the complex FFT, two copies in each graph (raw, fft, fist integral, second integral) for a minimum of about 9 copies. So maybe 30 copies would be reasonable to expect if the code is not optimized. It is getting up there!
simulation wrote:I need xy graphs, because I must detect breaks/gaps in time.
How do you fft it then? piecewise? You can do gaps in a waveform graph by graphing NaN for invalid points and they will not be plotted. What kind of breaks do you have? (large missing chunks? a few missing points only?) How do you deal with them?
Sending 770000 data points to a graph that is maybe 1000 pixels wide seems like a waste. How is the memory use if you clear the graphs and don't wire them to the graphs for testing? Are you using plain xy graphs or the express VI "built xy graph"? If so, do you have it set to retain data and how often do you reset?
Is your data in plain double arrays or something fancy such as dynamic data?
Have you tried the "show buffer allocation" tool? How often does the size of the array change? Are you building arrays in uninitialized shift registers?
How much RAM do you have?
Some useful information:
Memory Management for Large Data Sets
How Much Memory can LabVIEW 32-bit or 64-bit Use?
It would really help if I could see some actual code. Thanks!
Do you have a small datafile to read?
In "open_csv" alone, You are creating way too many copies of the file data in memory. The compiler is smart enough to avoid some of these copies (shown crossed out below, judging from the buffer allocation display), but the memory use is still significant.
Let's start with the small case structure:
You are reading the entire file as a string and pushing that entire data structure through an output tunnel of the case structure (first data copy!) where you process it so only the first line is parsed into five fields.
Inside the case structure you are branching the string (second data copy!) to process it.
The resulting 2D array (third data copy!) is then trimmed using "delete from array", making the fourth data copy!
Transposing create a fifth data copy, and a sixth data copy is created at the 2D array output tunnel. Now you are slicing out two or three rows (seventh data copy) and allocate a new buffer at each output tunnel (eight data copy) and placing it into the indicators (ninth data copy!).
As you can see the subVI alone carries about seven copies of the data in memory due to inefficient programming. This is way too much!
Suggested code improvement:
Read the first two lines using read lines and use "index array" to get the first line to parse the header. (this is a very small string, so we don't need to worry about its allocation). Keep the file refenum and wire to a plain "read string" To get the rest of the data with the first two lines already missing. Don't transpose, simply index out the columns later by wiring to the lower index terminal.
You should also make sure that the front panel of this subVI is NOT open when you run the program, else you get additional copies of the data for the indicators. (You can eliminate this overhead also by setting subroutine priority).
Why are you wiring the path reference instead of the value? You should also use path data, not strings for paths. You might be able to inline this subVI after correcting some of these things.
I have not looked at the rest of your code....
- last edited on by
Content Cleaner
@altenbach wrote:
You should also make sure that the front panel of this subVI is NOT open when you run the program, else you get additional copies of the data for the indicators. (You can eliminate this overhead also by setting subroutine priority).
Why are you wiring the path reference instead of the value?
OK, the presence of property nodes (for the name value) forces the subVI front panel to be in memory even if the front panel is not shown. Since the indocators contain huge arrays, this is expensive. Replace the reference with a path control and simply wire the path instead and eliminate all property nodes in the subVI.
Quote from here:
If the front panel of a subVI is not going to be displayed, do not leave unused Property Nodes on the subVI. Property Nodes cause the front panel of a subVI to remain in memory, which can cause unnecessary memory usage.
Also use path controls/indicators, not strings! There is even a primitive for "Not a path", simplifying some of your string gymnastics. 😉
Using strings for paths has two major flaws in your case that can lead to very unexpected behavior:
- paths displayed as strings are OS specific, so your code will break when going to a different OS (Win, Mac, linux).
- Your "<Kein Pfad>" will not work if the OS is in a different language. On a US computer you would need to compare with "<Not a Path>", a completely different string!!!
