low performance on 2d array operation

[MF0088]

I would like to filter out erronous values from a "large (slightly more than 2000*2000) 2D array. These value emerge from a CCD sensor wich can produce very values on a single pixel.

It is easy to detect. My goal here is to replace these values by the average of the surrounding 4 pixels if the value is larger than a specified value (most of the time, the CCD is saturared so it would be the maximum of an N bit integer).

I am trying to run a 2 nested for loops, scanning every value of the array, testing and if required, replacing. I have a very low performance after a few iterations. I am most certainly doing something wrong but I never worked with large arrays before, so I'm probably not using the correct tools.

I am using Labview 2015.

[crossrulz · Accepted solution]

Don't use the parallelism on your FOR loops.  Use Shift Registers on your FOR loops.  This way you are working with the array in place.

Other comments:

No need for the timeout.  Delete it.

Make an event case for the Stop button's value change and put the stop button inside of it.  That way you can properly stop the loop without polling.

[aputman]

Are you getting the output that you expect?

[Alexander_Sobolev]

Does this vi work and really replace values? It looks like its output is always last iteration -  bottom right corner of the picture. Where would result of inner iterations go? 

Strange that labVIEW does not recognise this and executes all iterations, not only the last one.

It generates a number of loop instances (each one needs a copy of the array - here where all your time goes) then replaces one element in this array (or leaves it as it is) and discards the result.

As crossrulz said, remove parallelism, use shift register for main array.

May be it will be better to put comparison into the loop to avoid generating the same size boolean array.

You know your algorithm has troubles for bad points on the border and adjacent ones?

[altenbach · Accepted solution]

In addition to what tim said:

• Yes, use shift registers instead of concatenating tunnels.
• How many time do you think you need to +1 or -1 the same number in parallel? Do it once and branch after the operation instead!
• Definitely remove parallelization on the inner loop, parallelizing a parallel instance is not a good idea. It should also be removed from the outer loop, I think. You cannot parallelize once you use shift regsiters, because you are changing elements and the outcome might depend on the order of execution of the parallel instances. Something you cannot control.\
• I think using concatenating tunnels on the otherwise resulting 3D arrays is a big waste of memory. Do you get the expected result? Have you looked at the output sizes? (Seems you get NxM 2D arrays of the original size concatenated, i.e. orders of magnitude more elements!)  Most likely, your code is slow because you are creating these gigantic arrays. Operate in-place on the 2D array. No extra memory needed.
• Why is there no event for the stop button?
• You never look at the modified 2D array, so all calculation seems pointless! Is there more to it? (no indicator anywhere!). Maybe the graph terminal belongs in front of the event structure?
• What's the point of the small orange wire in a big blue ocean of wires? Finding Nemo?
• "Index...replace" is easier done using an IPE structure.

[altenbach · Accepted solution]

Since you need to create a new 2D array to use only unchanged, original pixels for the avaraging, I would just autoindex. See if this is any faster.

Do you have an example file?

(Code not tested, so make sure it works correctly...)

(EDIT: Sorry, I had the indices swapped incorrectly. I now attached a corrected version)

[MF0088]

Indeed the operation is done in 2-3 seconds on my desktop computer using shift registers.

Thanks for you help here.

I know the code is abolutely not optimized regarding boundary conditions, if 2 adjacent pixels are deffective and number of times I call Increment or Decrement, but it was a rapid VI I made on a test file that could not fail these tests. I will work on that but I know how do it it.

For what it's worth, here is what I have now, as a working solution.

[altenbach]

• You still have several coercion dots, indicating type mismatches.
• If you want array elements in order, starting with zero, you don't need to wire any indices of index array.
• Since you are autoindexing (on the green array), there is no need to wire the N of the loops.
• Have you tried my solution above? (note that the first attachment was incorrect, so please try again, see above)
• How many bits are in the detector. I am sure that I64 is overkill. Do you really have possible negative values? Maybe U8 or U16 is all you need.

[MF0088]

You solution works indeed, probably the most efficient yet. I have to try on real data but that's up to the experiment to work.

My detector gives me  unsigned 16 bit values but the API from some instruments uses I32 so that's what I'm using.

[altenbach]

---

@MF0088 wrote:

You solution works indeed, probably the most efficient yet. I have to try on real data but that's up to the experiment to work.

My detector gives me  unsigned 16 bit values but the API from some instruments uses I32 so that's what I'm using.

---

The "Out" array in your picture is I64 for some reason.