Best way to transfer large datasets between actors (Actor framework)

[JonasCJ]

Hi everyone

I am in the planning / design phase of a larger application for which I wish to use the Actor Framework. It seems like a good "design pattern", easy for multiple developers to work on, easy to expand and easy to maintain.

In this application I need to transfer data betweeen actors/modules at a considerable rate. The data is the continous measurements from a DAQ device (or multiple devices) sampling at 200kHz. That's 200 kHz * 8 byte = 1.5 Mb/s per device.

There is no way this is done using the messages of the actor framework - those messages are designed to signal start, stop, select tasks etc. within the application. So how to transfer that amount of data between actors?

I've thought about using TCP/IP on localhost - this could also easily be expanded to allow different parts of the program to run on different machines on a network. I've even tested it and it works quite well. But I would like to hear some opinions or alternatives before I decide to go with this solution. Any thoughts or alternatives?

Thanks

Jonas

[tst]

Well, it highly depends on how you're going to use the data, but for a large data set, you will probably want to minimize data copies, and the obvious candidate for that is a DVR, so you create the DVR, then pass it to the actors using a message and let each actor act directly on the data.

If you only care about streaming the data, then you can do the same thing with a queue. If you only enqueue and dequeue data, then LV is smart enough to not need any reallocation. It just takes the pointer for the enqueued data and gives it to the wire on the dequeue end.

Another alternative is to keep the data in a single processing actor and have the other actors send it messages asking it to process the data.

[JonasCJ]

Thank you for your reply tst.

I will definetly look into Data Value Reference (I haven't really worked with them before). So the way I use my data is by taking one chunk of samples at a time (e.g. 50k sampels), do for example an FFT of this chunk and display the result on a graph. Then the next chunk and so on. In a sense I would say that I never have more than a few of these chunks active / laying around. When the fft is made, the data is written to disk or similar the data is no longer needed.

Will DVR work for having an array (representing an FFT of a given time slice of samples) "stored" locally in one actor and displayed (e.g. graphed) in another actor? This whole display business sounds more like copying than manipulating data inplace (which is the purpose of DVR, as far as I know).

[tst]

The display will definitely require a copy in any case, but the basic answer is yes - the DVR will let you access the data from more than one place (although, again, taking it out of the IPE structure and into the display will cause a copy). Also note that graph indicators don't always place nice with large amounts of data and you might have to decimate it first. Stress testing an experimental version before you fully commit is recommended.

If you really do have only some of the data lying around (let's say 1.5 MB/s * 40s = 60 MB) that's not so bad even if the data is in a single array, but certainly if it's in multiple arrays where it doesn't need to be in a contiguous area of memory. In such a case, you will probably want the actors to process the data into standard form and not pass around DVRs to raw data, simply because that requires the actors to play by the same rules.

[JonasCJ]

So what do you think of using TCP/IP to pass data from actor to actor locally? You said it could be done with a simple queue. What I've tested is basically two queues each with a producer/consumer loop:

Actor 1
Local queue 1
Producer: the DAQ device and sampling VIs (samples the DAQ and enqueues new sample chunks on Local queue 1)

Consumer: TCP/IP transmission (dequeues sample chunks from Local queue 1 and transmits them via TCP/IP)

Actor 2
Local queue 2
Producer: TCP/IP receiver (receives and enqueues the sample chunks on Local queue 2)
Consumer: data processing, graph display etc.(dequeues sample chunks from Local queue 2 for processing)

Could this be done without TCP/IP and just a LABVIEW queue? Passing the queue in a message from one actor to another?

[tst]

---

@JonasCJ wrote:

Could this be done without TCP/IP and just a LABVIEW queue? Passing the queue in a message from one actor to another?

---

If it's in the same process then yes, and it would be more efficient, because with TCP you have to create a copy when converting the data to a string, another copy when receiving and a third one when converting back. Unless you really are planning on distribtuing it, I would suggest avoiding that. Sidebar - AQ has been talking about networked actors and I believe there should even be an experimental branch in the AF group in the community area. You might want to check that out.

Incidentally, if you are planning on splitting it and don't actually need the real time processing, you might consider saving the data to a network drive and then simply having the other side loading the data from the file. It should be simpler.

[JonasCJ]

Yes I know about the overhead of the TCP connection - I have made a small proof of concept. It's not too cpu intensive, but of course another method might be less intensive.

Just to make sure: when you say "same process" ("If it's in the same process then yes") you mean if the two VI's/actors gets compiled into the same executable?

[tst]

---

@JonasCJ wrote:

Just to make sure: when you say "same process" ("If it's in the same process then yes") you mean if the two VI's/actors gets compiled into the same executable?

---

Yes. More accurately, a queue reference (like a DVR and a notifier and other references) is functional within an application instance. In the case of an executable, that's the executable. In the IDE, there are a number of things which can create an app instance, but for practical purposes just think of every target in a project as a separate instance. You can see (and actually change by right clicking) the instance each VI is loaded in in its bottom left corner.