Code structure for data gathering—ways to avoid dynamic array size?

[vekkuli]

Hello! Yes, time to milk out some valuable free advice once again.

I'm working on a vision tracking algorithm and I've come to worry about the way I initially set up the data gathering. I suppose there are various different schemes I could go with but most of all I would like hear what you guys would suggest.

Here's what I have then: A 1D array of coordinate clusters represents the points to track. These are affected by the tracking algorithm but points can also be removed if they hit the boundaries and new points can be introduced. A 2D array of coordinate clusters represents the accumulated tracked points. When a current coordinate meets a boundary the point is deleted from the 1D array and the appropriate column is removed from the 2D array. These removed columns represent the gathered data units in a desirable spatio-temporalish style. See the attached VI for "clearance". I made the VI to illustrate the case, the real project with all the vision hocus-pocus is a formidable spaghetti universe.

Now, the issue is that in large scales I can see the constantly varying 2D array size ending up being a major resource hog. What sort of structural alternatives would you suggest to the producer bit? Should I use an initialised 2D array and deal with the data in chunks? Should I deal with the extraction in another loop altogether? I don't need to use the gathered data in the same context so writing to a file could be put to use here as well.

Thank you in advance!

[M.athieu.]

Hello,

I'm working on LV2009 can you upload your VI in this Version ? Thanks 🙂

[Neos]

can't see your vi ( have LV 2012 ) but some things are always true for every version

If you are resizing the 2D array every time ( say you are using Delete from array primitive ) it will cause the array to switch over a new place in memory so in the process yes there will be a performance hit.

so intialising the array and using function to replace older element with new one ( say 0 ) is definetly the one most thing. 

Extraction in other loop will help if you want to speed up your data gathering loop but it depends on which speed you are getting data and how much time the processing is taking.

[vekkuli]

Yes, of course, should've thought of that. Me and my fancy-pants LV2013 😄  

[Neos]

As said eariler bulid array and delete from array........

Do you know maximum size of your arrays??????

for the elements you don't need put say 0 and don't respond to those values

[vekkuli]

Yes you are right, the Build Array and the Delete From Array are the key issues in my opinion as well. The maximum size of the arrays is unknown thus I cannot just straightforward initialise the array and use replacing instead. There could be a good way to use initialised array anyhow but then there should be a whole different way to extract the ready parts.

[Neos]

Instead of doing this operations every time  you should follow an something in which reallocation can be minimized

Darren has one nugget on this topic ......search it on forum

[aputman]

Instead of deleting the items from the array, is it feasible to add a "out of bounds" flag and a "time" constant to your cluster and only process the ones that are still in bounds?  I don't know what your consumer loop is doing to the data but after the for-loop is completed, your array could then be processed. 

[vekkuli]

---

@aputman wrote:

Instead of deleting the items from the array, is it feasible to add a "out of bounds" flag and a "time" constant to your cluster and only process the ones that are still in bounds?  I don't know what your consumer loop is doing to the data but after the for-loop is completed, your array could then be processed. 

---

Interesting idea, I could possibly keep the data structure more flat with such methods. I wanted to implement the producer-consumer structure as I thought the application might be running and gathering data for hours. That means there might be close to million iterations in total, some hundreds of objects in the end. Thus the idea to pop out the finished objects and maybe write to a file in the consumer loop, for instance. Although, who knows, maybe the scale isn't all that large after all and I might as well just ditch the whole queue.

---

@Neos wrote:

Instead of doing this operations every time  you should follow an something in which reallocation can be minimized

Darren has one nugget on this topic ......search it on forum

---

Minimizing the reallocation, that would be great of course. I looked at the nuggets, I'm not sure which one you ment. I found one where he introduced a way to deal with ragged array -style data structure, which in this case would, in principle, be very convenient as the object columns in the 2D array, the data units, are not the same length. You'll find a lot of leading zeros. However, for all that I know it wouldn't offer much performance-wise. Was it another nugget you were referring to?

[johnsold]

I have been thinking about your issues and have some ideas and questions.

As you have considered, Build Array and Delete from Array are very large resource hogs when the arrays get large. Initialize Array and Replace Array Subset are your friends.  For large arrays they can be an order of magnitude faster.

I also like the idea of the out-of-bounds flag.  Using NaN (Not a Number) is one possibility.  A primitive comparison function for Is NaN? exists and you are not doing any calculations on the points/tracks in your example code.

I think there is a better way to organize your tracks, but I have not figured out what it is yet. For one thing you have redundant information in the 2D array. A lot of it when the data sets get large.

Do you have any idea of what is the largest possible number of Points that you will be tracking at any instant of time? I suspect that it is a small fraction of the  number of pixels in your images, although that could still be a fairly large number. What is the largest number of steps in any one track? Can a Point "circulate" indefinitely or does it approach a boundary in a known time limit?  Do tracks ever intersect, cross, or merge? How do you want to handle those situations?

Lynn