FGV vs Global Variable vs DVR

[Kiwi6]

Hi,

I'm working on a big project and I'm stuck on how to implement a efficient way to distribute data in my evaluation VI.

To give an overview of the problem:

I have a measurement device with several sensors (temp, voltage, distance, angle,...) which produces a data file which is read by my LabVIEW code. Each of the sensors produces double precision values in the order of 1,6 million data points over one measurement. For some of the sensors I have more physical sensors installed (like up to 6 distance sensors all producing the same amount of data).

Also the system should remain expandable, maybe there are more sensors in the future. For now the only restriction I will place is that each sensor produces the same amount of data points as all the others in double precision, so no weird cluster data or something like that.

As for function, I would like to write the data once to this "data distribution structure" at the beginning when the VI is called and subsequently only read from this memory location (and do calculations based on the data, but not overwriting the data in memory).

Since I'm working in a 32 bit version of LabView (which I can't change to 64 bit for legacy reasons) I have to be careful with memory management.

So basically for now I want to create an array for each sensor and store all the data for this sensor in this array, to make sure I don't allocate too big chunks of memory at each time.

The question is now how to distribute the data efficiently:

I would naturally use a FGV, it is jut the first thing that comes to my mind. However, other than the data origin, all the FGVs would be structurally the same. Also, I don't foresee any of the data types requiring action, that the others wouldn't require.

So I thought about using something similar to the code in the This post.

So I open a reference for each of the data arrays need and keep accessing the specific data buy using the reference I created for this data.

However, to access the data by reference I need a type specifier wired to open reference (to get the strict reference by reference requires). Since I create the type specifier by browsing for my "master FGV" VI, I'm a bit worried this uses absolute paths internally and the type specifier will loose the connection to the MasterFGV VI once I deploy everything. Would this happen or does LabVIEW know what I was selecting during development based on the location of the VI in the project hierarchy?

The other thing I was shown was DVR.

I would create a DVR for each sensor type, write some code to access the correct reference once I need data for a specific sensor and read the data from the reference and output it. However the only thing that come to mind for reading a reference is this:

Is there a best practice for reading data from a DVR without manipulating the data in the reference? Also, will this create a copy of the data in memory? Everything in me says it will, since LabVIEW probably can't be sure I won't change the data that was referenced in the first place.

The last idea I had was using a global variable as I only write once and read multiple times throughout the code. However, having the data encapsulated just seems so much neater and is less open to misuse by me or others in the future.

Which of the three options would be best, or am I missing some fourth, completely obvious implementation that would be so much easier than all of the above? Thank you for any comments in advance.

Code is written in LabView 2017, 32 bit and deployed to a 64 bit machine running Win10.

Cheers, Kiwi

[mcduff]

Attached is code that should run in Windows, 32 bit LabVIEW only. The dll calls seem only to work in 32 bit LabVIEW.

Anyway if you go the DVR route, keep the wire in the in-place structure. Once you branch it outside you appear to get a memory copy. According the dll counters, when keep the wire inside you do not get a copy.

mcduff

[Kiwi6]

Hi mcduff,

Thank you for your answer, it did clear a few things up for me.

The reason I would use this version to branch outside of the in-place structure is basically that there is a lot of code already in place that is expecting its data to arrive as a 2D array. So to keep everything in the in-place structure, I would have to rewrite a considerable part of the project. I will think about it however.

Follow up question, what happens if I put a SubVI in the in-place structure to do something with it? My guess is, I would still get copies of the data regardless and to make it efficient I would have to modifiy the SubVI itself, do the data access and calculations in the code of the SubVI in a giant in place structure and use the results further on?

Cheers, Kiwi

[mcduff · Accepted solution]

---

@Kiwi6 wrote:

Hi mcduff,

Follow up question, what happens if I put a SubVI in the in-place structure to do something with it? My guess is, I would still get copies of the data regardless and to make it efficient I would have to modifiy the SubVI itself, do the data access and calculations in the code of the SubVI in a giant in place structure and use the results further on?

Cheers, Kiwi

---

Whether you get copies will depend on your code. What is the product of your code, is it a single number or an array of numbers to be used elsewhere.

Below are some screenshots where I use subVIs within IPE structures. Sometimes there is a copy as the subVI needs a copy, sometimes there is a copy because of parallel iterations (each one needs it own memory space), and sometimes there is no copy. Mileage varies, best recommendation is to test and test. Sorry I cannot offer better advice.

mcduff

[crossrulz]

Does all of your code only care about the latest readings or all of the readings?  I ask because Globals, FGV, AEs, and DVRs are types of Tag data, where you only care about the latest data and any old can be thrown away.  If you are constantly adding to your array, you will have some major memory issues, so I hope you are not doing that.  So if all of your other modules need to process all of the data, preferably as it comes in so you can limit memory usage, then I would use Queues (1 queue for each receiver) or an User Event (1 event can go to multiple Event Structures) to pass the data to everybody as it comes in.  This is a variant on the Producer/Consumer.

[Kiwi6]

Hi

@ mcduff:

I do mostly simple evaluations. For example with the distance data I look when the distance to the target drops below a threshold I look for the minimum in this area and with its location i extract data from the other data sets, for example the angle at minimum approach. The spacing of the minima also tells me how far along in the process I am as the device measures the movement of the measurement device through a industrial plant.

Those results will be stored in a result array which is an array of clusters containing a data point each (minimum distances, angles, temp and voltage, measurement index of the minimum, other values calculated based on these). But this array is normally no trouble, only a few 100 elements in length so no need to be extremely sophisticated with data storage.

Thanks for the screenshots, that looks like something I can try.

The advice is still sound, I will try the suggestion and report back on which solution I will finally choose.

@ crossrulz

If I do it right there should be only one write operation for each sensor type in the program flow and subsequently I'm just reading and evaluating the data. So "old" data is something that should not happen in the case of this program, all data should be up to date once I need in it in the course of the program.

Adding to an Array constantly is what I'm doing now (As I throw all the data of all the sensors into one array, and add to it once I do the conversions of digits into distance units and degrees. At least those I need to do only once and all the code after works with converted measurements) and why I have major memory issues and I'm trying to distribute the sensors to arrays or other kinds of data distribution. I found out the hard way, that it is a bad idea to be laissez-faire with memory and just scale up code for big data sets which is meant to be used in small data collections.

My data is read from file, so I do my evaluations after the measurement is completed. Therefore I don't think Producer/Consumer is the design pattern for my application.

[crossrulz]

---

@Kiwi6 wrote:

My data is read from file...

---

There is the kicker.  You should read in smaller chunks, pass that to all of your processing, let the processing happen, then read the next chunk.  This is sounding more like a simple While Loop and/or State Machine application now.

[mcduff]

@crossrulz is giving good advice, if possible try to break down your processing to chunks, that way less memory.

When you go down this road, you will have to decide on ease of programming, debugging, speed, and memory usage. You will have to decide for each case which is more important. Look at the following code (LV2017 attached)

1. Initial Memory allocation 72MBytes for the array. Seems like 8MBy too many. But the inner for loop uses 8MBytes, while 8 million doubles is 64MBy, for the total 2d array.
2. The simple Mean primitive uses an additional 8MBytes of memory, but is clean, easy to read, debug etc.
3. The in-place mean does NOT use any additional memory, but more work to code, possibly debug, read, etc. May be slower, did not time.

You can enable/disable the loops and use the "Profile Performance and Memory" tool to look at the memory allocations.

mcduff

[mcduff]

---

@mcduff wrote:

@crossrulz is giving good advice, if possible try to break down your processing to chunks, that way less memory.

When you go down this road, you will have to decide on ease of programming, debugging, speed, and memory usage. You will have to decide for each case which is more important. Look at the following code (LV2017 attached)

1. Initial Memory allocation 72MBytes for the array. Seems like 8MBy too many. But the inner for loop uses 8MBytes, while 8 million doubles is 64MBy, for the total 2d array.
2. The simple Mean primitive uses an additional 8MBytes of memory, but is clean, easy to read, debug etc.
3. The in-place mean does NOT use any additional memory, but more work to code, possibly debug, read, etc. May be slower, did not time.

You can enable/disable the loops and use the "Profile Performance and Memory" tool to look at the memory allocations.

mcduff

---

EDIT: Just tried running in parallel, much larger memory saving, and faster for the IPE structure.

[Kiwi6]

It took me a while to implement this.

I went with DVR, as suggested. Each sensor gets a DVR, I automated the set up of everything. Sensors are then grouped by type, and all the references are stored in a global variable. So now I can get the data of one sensor or one type of sensor, depending what I need.

As for the code, I work in IPE structures where I can and pull the data out where I can't. Not the most efficient, but it is ok for now and I don't need to re-write stuff that already works.

The data is still read as a whole, but as 8bit Integer and I use the minimal data type where I can and convert to double only later on where the precision is needed.

Thanks for you help, it was truly appreciated.