Reduce System Memory Usage While Saving Waveforms

[nickb34]

I have created a VI that captures timing sequence data for up to 20 scope (digitizer) channels. When saving large waveforms to .CSV, the memory usage will continue going up every time I save a new file. I'm wondering how I can optimize my VI to return to the memory usage level before it was called. I built a quick top level VI to simulate this problem and have attached both files. I appreciate any help, thanks!

[Bob_Schor]

You do realize, don't you, that you are trying to write out a text file that has 24 channels of a "waveform" containing 3,000,000 points, don't you?  If we assume each element takes 10 bytes, that's 72 million points that have to be converted to a Delimited Spreadsheet formula.

Also, you call what you are saving a "waveform".  Did you know that "Waveform" has a specific meaning in LabVIEW?  It looks like a Cluster, but isn't a Cluster.

I modified your code to construct Waveforms, 24 channels of 3 million points each.  I sent this array to the Waveform function "Write Waveforms to File" -- the file was around 560 MB.

Note that if you write this as a Delimited Spreadsheet, it will have 24 rows and 3 million columns, and be ridiculous to process.  You could transpose it, but I'm not sure that would be a good idea.  Maybe you should rethink what you are trying to do ...

Bob Schor

[altenbach]

Not sure why you have two loops, it all could be done in one.

Does each plot really have a different t0? In the subVI you are throwing most of them away.

Why do you have that "request deallocation" VI?

Do you have the subVI front panel open or closed? That 2D output sure will take a lot of memory.

Your while loop in the subVi is really dangerous, because it will never stop if the input is e.g. zero by accident.

Did you notice that you are missing the first time point?

[nickb34]

Bob,

I definitely realize what I am trying to do. The output file has the potential to be huge. The concern here isn't that it creates large files, the concern is that it consumes a lot of memory after the call has been made and that memory isn't recovered after running the VI.

This waveform is the data type that the "niScope multi fetch cluster.vi" generates automatically.

As written, the code generates a delimited spreadsheet with 25 rows and 3 million columns.

I'll try the "Write Waverforms to File" which I discovered after writing this post. Though when I looked at it, it was actually fairly similar to what I did on my own.

[nickb34]

altenbach,

1. I see that now, I just threw something together real quick to simulate what my larger VI does.

2. They all have the same t0

3. This was my attempt at mitigating the issue that I'm asking about here. I thought it might be a memory deallocation problem

4. subVI panel is closed, yes it can take lots of memory

5. I see your point here, while this vi should never be able to be run if there isn't a waveform, I can see where changing this to a case structure with a default would be safer

6. Good catch, I'll fix that

[mcduff · Accepted solution]

The problem with your VI is you are trying to do everything at once; that creates multiple memory copies.

You need to write in chunks of data.

Here is a start below, the block diagram can probably be reduced to the size of a small FedEx package.

2018 Version saved.

You need to add the part for the time array, I don't have time to do that. This should use less memory; play around with the chunk size, larger chunk more memory but faster, smaller chunk less memory but slower save.

mcduff

[nickb34]

Thanks for this. I'll have to add in some logic for if the data set is less than the chunk size and some things like that, but initially this looks to help me out with what's going on. I'll put it through the paces for sure.

Is there an advantage to using write to file versus the write to spreadsheet?

[mcduff]

---

@nickb34 wrote:

 I'll have to add in some logic for if the data set is less than the chunk size

---

The logic is already there, the For Loop will run once unless the number of points is 0. Test it.

---

@nickb34 wrote:

Is there an advantage to using write to file versus the write to spreadsheet?

---

You have a bit more control as to what you want to do. The higher level VIs are easier to use, but may do things you don't necessary need, among them make data copies.

mcduff

[altenbach]

---

@nickb34 wrote:

altenbach,

1. I see that now, I just threw something together real quick to simulate what my larger VI does.

2. They all have the same t0

3. This was my attempt at mitigating the issue that I'm asking about here. I thought it might be a memory deallocation problem

4. subVI panel is closed, yes it can take lots of memory

5. I see your point here, while this vi should never be able to be run if there isn't a waveform, I can see where changing this to a case structure with a default would be safer

6. Good catch, I'll fix that

---

You don't really need a loop. for inputs <1, here's what you could do (or similar).

You know, waveform graphs also accept clusters of [x0, dx, 2D array]. Works well if all traces have the same x values as in your case.

As I said, make sure to disable debugging and keep the subVI panel closed. Does it really need to be a text file? A binary option would see so much more reasonable for all that. 😉

Here are a few quick mods that can also give you some ideas. (Of course saving in chunks might still be needed)

[nickb34]

---

@altenbach wrote:

---

---

You don't really need a loop. for inputs <1, here's what you could do (or similar).

---

 That's a fancy algorithm, I'll have to look into it and see what's happening...

I came up with a solution and it seems to be substantially faster, as well as no longer having the memory issues that I previously experienced. I'm sure it could be tweaked to look better, so I may work on it a bit more. I think the chunking was necessary and appreciate the perfect primer from @mcduff.