LabVIEW
Data copies and circular buffering
I am working on a circular buffer implementation, and in the early phases of testing my approach I noticed something puzzling. I am trying to store LOTS of data in the buffer. I was using a hard disk circular buffering scheme with files, but I want to give the HDD a break, so I'm trying to do it with RAM.
The problem is that I cannot seem to get less than 3 data copies of the entire buffer. I could sort of understand how there may be two, but I'm trying to do everything as in-place as possible. Even when I show the buffer allocations, I can only see two black dots associated with the buffer (a 2D array of SGL).
It get's weirder....
I am looking at the Memory Usage page of the VI Properties dialog. If I initialize my buffer with a 1,000 x 1,000 array of SGL, I would expect roughly 4 MB for each copy of the array. Depending on the presence of a local variable, I can get close to 12 MB or close to 16 MB. That means that there are either 3 or 4 copies of the buffer floating around. What really gets me is that the local variable actually results in the 3 copy version! When I add a read local variable as the initializer to the shift register, there are 3 data copies of the buffer based on the memory.
I have attached the VI. If anyone can explain what's going on or suggest an alternative approach to the buffer, I'm all ears.... er eyes. The function runs extra fast when put in subroutine priority. Speed is not the big issue here, but memory footprint is.
Thanks,
Dan Press
PrimeTest Corp.
www.primetest.com
The problem is that I cannot seem to get less than 3 data copies of the entire buffer. I could sort of understand how there may be two, but I'm trying to do everything as in-place as possible. Even when I show the buffer allocations, I can only see two black dots associated with the buffer (a 2D array of SGL).
It get's weirder....
I am looking at the Memory Usage page of the VI Properties dialog. If I initialize my buffer with a 1,000 x 1,000 array of SGL, I would expect roughly 4 MB for each copy of the array. Depending on the presence of a local variable, I can get close to 12 MB or close to 16 MB. That means that there are either 3 or 4 copies of the buffer floating around. What really gets me is that the local variable actually results in the 3 copy version! When I add a read local variable as the initializer to the shift register, there are 3 data copies of the buffer based on the memory.
I have attached the VI. If anyone can explain what's going on or suggest an alternative approach to the buffer, I'm all ears.... er eyes. The function runs extra fast when put in subroutine priority. Speed is not the big issue here, but memory footprint is.
Thanks,
Dan Press
PrimeTest Corp.
www.primetest.com
A few very quick tips after looking at the diagram for a few seconds (I'll study it in more details later):
(1) Delete that local variable, it is not needed (leave the shift register uninitialized, it will do the same thing!).
(2) Set the VI to subroutine priority. This will avoid the extra data copies in the FP objects (I think) and also remove some of the call overhead.
(1) Delete that local variable, it is not needed (leave the shift register uninitialized, it will do the same thing!).
(2) Set the VI to subroutine priority. This will avoid the extra data copies in the FP objects (I think) and also remove some of the call overhead.
Thanks for the response!
I noticed that adding the local reduced the number of data copies from 4 to 3. That's one of the things I'm asking about. I wouldn't normally put it in there, but I saw that the buffer allocation indicator was lighting up on the shift register and on the indicator. The local gives the shift register a data structure to bind with in memory (theory talking here).
I left it in Normal priority so it is easy for others to run. Subroutine priority takes away the run button.
I have tested the speed and that is not the issue. I'm worried about memory footprint on this one. I want to have as large a buffer as I can. RAM = time storage in this case.
Thanks,
Dan
I noticed that adding the local reduced the number of data copies from 4 to 3. That's one of the things I'm asking about. I wouldn't normally put it in there, but I saw that the buffer allocation indicator was lighting up on the shift register and on the indicator. The local gives the shift register a data structure to bind with in memory (theory talking here).
I left it in Normal priority so it is easy for others to run. Subroutine priority takes away the run button.
I have tested the speed and that is not the issue. I'm worried about memory footprint on this one. I want to have as large a buffer as I can. RAM = time storage in this case.
Thanks,
Dan
You don't need any local or global variables to get the memory down to 7815k.
Here's how:
(1) Place the Data_out terminal inside the "read" case.
(2) remove the data_out local variable, it no longer makes a difference.
Of course you should be aware that you should not use the data_out output unless the mode is set to read.
Here's how:
(1) Place the Data_out terminal inside the "read" case.
(2) remove the data_out local variable, it no longer makes a difference.
Of course you should be aware that you should not use the data_out output unless the mode is set to read.
There are three buffer allocations, but the one for Data IN shouldn't be very big, it's an empty array by default.
This is a neat discussion. We're always harping on optimization, and I don't think you can get more optomized than those simple examples. I guess the main question is why do we need 3 buffers for a control / indicator?
Dan Press
www.primetest.com
This is a neat discussion. We're always harping on optimization, and I don't think you can get more optomized than those simple examples. I guess the main question is why do we need 3 buffers for a control / indicator?
Dan Press
www.primetest.com
