Hello, I would like to use Labview 2020 64-bit Python node to proces a stack of images (tens of 2048 x 2048 16-bit images) previously acquired by a camera in Labview with some custom ML Python code. I could write a test program that runs successfully and pass the images as a UINT16 3D array to the Python node but the memory transfer from Labview to Python is painfully slow. For instance, passing a modest stack of 10 images (80 MB) takes over 13s! I do not know what kind of memory copy / data conversion is performed by Labview under the hood but it is in practice infinitely faster to write the images to disk and read them from the Python function... I would like to avoid disk transfer and I think that this kind of Python ML application might be useful to other people. Is there a way to achieve a faster memory transfer? Note: Fiddling around, I could achieve over 4x speedup by reshaping the 3D array to 1D before passing it to the Python node but it is still order of magnitudes slower than the speed I would expect on this computer (Intel i9-10900X 3.7 GHz, 128 GB RAM, Windows x64 10 Entreprise). Best, Sébastien

A few random thoughts from someone who hasn't used the Python nodes much: -Are you receiving back a 3D array? Could you try the "flatten to 1D" trick for received data as well? In other words- are you sure it's the "getting data into Python" part that's slow, not the Python part itself or the return from Python? Oftentimes you can speed up large data transfers by preallocating arrays, but that's more on the LabVIEW side of things than the Python one. -The Help for the Python node mentions that, by default, arrays are converted to Python lists. You can right-click that and tell it to convert it to a NumPy array. I have no idea how Python handles lists, but I bet you it's not the same way that LabVIEW handles arrays. NumPy arrays might be better... who knows. Unfortunately you are probably having to step through each element of the array one by one and copy it from LabVIEW's memory into Python's memory. The Python node may not know the size of the array when it creates its own memory space and will have to allocate space, fill it, reallocate it, fill it, and so on. That's likely what's taking so long. Memory reallocation of massive datasets will nuke performance in LabVIEW, and it's probably the case in your example. See this post: https://stackoverflow.com/questions/311775/create-a-list-with-initial-capacity-in-python The top voted answer has some issues (for example, it only does 10k elements where you have 4 million per image), so review the second answer. That notes that Python lists have no built-in preallocation. NumPy does have preallocation: https://stackoverflow.com/questions/3491802/what-is-the-preferred-way-to-preallocate-numpy-arrays so that might help, but that depends on if the LabVIEW Python node preallocates NumPy arrays. Honestly I don't know. I would first try switching from Python lists to NumPy arrays to see if that helps. If that doesn't work, and the issue is indeed array reallocation, you'll need to switch to something like tcp. That way, you can load a session, preallocate your memory, then transfer your images via tcp into a preallocated space. Then again the bottleneck might not be preallocation, but that's where I'd start. PS: Assuming you have 30 images you'll have ~126 million array elements. If Python doubles its memory allocation each time, and it starts with 1 element, you'll have to reallocate memory 27 times, possibly copying the entire dataset each time unless it can allocate contiguous memory, which may or may not be possible.

LabVIEW

Very slow memory transfer when passing arrays to Python node

Solved!

Author

# array_to_numpy.py

import numpy as np

def array_to_numpy(A):
     An = np.sum(A, axis = 0).astype('uint16')
     return An

Message 11 of 11

(360 Views)

LabVIEW

Very slow memory transfer when passing arrays to Python node

Re: Very slow memory transfer when passing arrays to Python node