Actor Framework limited by not having an Actor Arbiter/Administrator

[AristosQueue (NI)]

---

@jlokanis wrote:

Allowing actors to direct message each other outside of the tree will not lead to race conditions as long as every actor behaves in a completely asynchronous fashion. 

---

I used to believe that, but I have seen counter examples to that in actual code.

Here's the situation that the programmer codes:

1. A sends X to B then sends X to C.
2. B receives X and decides to send Y to C.
3. C receives X, changes modes, then receives Y in the new mode.

Because of thread scheduling, here's what can happen rarely:

1. A sends X to B.
2. B receives X. In response, sends Y to C.
3. A sends X to C.
4. C receives Y while still in the previous mode and misbehaves. Then it receives X, sometimes compounding the error.

The fixes are many. Some options:

• have C recognize when Y comes out of sequence and store it until it receives X.
• have A grab some sort of semaphore that prevents anyone from sending anything until A has sent to both B and C.

But if the programmer never thinks of that reordering as possible -- and it took a long time to conceptualize what was happening even once the bug was identified -- then they don't program in such protections.

[AristosQueue (NI)]

Now, having said all the previous... Pitor asks about adding an arbiter that could check for these situations and control them, instead of encouraging devs to wander off into the wild on their own.

I've given some thought to that. It's not a bad idea in some ways, but I think the right answer is more along the lines of having the IDE facilitate creating specific patterns (like the pipeline loop) rather than creating an arbitrary by-name arbiter. Checking for synchronous messages on both sides of a connection is possible in theory with the VI Analyzer or with a run-time lint picker.

The debug facilities of AF could definitely use some strengthening. But I think those need to be through IDE injection, not through modifications of the framework. Framework mods like the Monitor Actor or the DETT messages have had a history of destabilizing working code or introducing performance bugs. They're both better than what we had before, but not really good solutions.

Rewriting the AF using channel wires has given me some ideas in this domain, but nothing concrete yet.

[jlokanis]

AQ: those are good points.  And I am aware of that situation.  In my applications, I maintain state that more than one actor needs to utilize in singleton objects.  This allows me to protect any writes to that state from getting out of order.  If an actor needs to respond to a state change, the actor that made the change can simply message listeners that a change has been made.  The listeners are free to respond to that change at any time in the future, even after multiple changes have been made.  Im my applications, state change is always lossy and I am fine with that since it is mainly used to update UIs.

Data, on the other hand, is generally not modified in more than one place in my application and is also stored in singletons but with only one actor making changes while the rest simply read.

Crossing the network boundary is an issue of course since we cannot share a singleton outside of one application instance.  In those cases, any data changes are done through tagged structures (built on variant attribute trees) to allow for out of order updates to be applied and still result in the correct end data set.

It is all rather complex but as you point out, complexity cannot be avoided in systems like this.

My main concern with the AF tree really boils down to performance.  All that messaging up and down the tree seems like we are adding layovers to something that would be much faster if it was a direct flight.  That might be fine in a system where you are performing a single set of tasks (like testing a single DUT).  But in my system, I multiply that by around 500x so every little bit of performance degradation counts.  I need to be as direct and lossy as I can with state changes and burden the CPUs as little as possible.

Maybe I am wrong and the AF tree would not be as much a burden as I suspect.  Unfortunately I would have to spend a few years rewriting my applications in AF to find out.

If I was writing a one-off app with some parallelism but not on the massive scale I deal with now, I would likely use AF and I do not doubt I would be successful.  I would really like having access to the automation tools it provides that I end up building myself for my architecture.

[AristosQueue (NI)]

> All that messaging up and down the tree seems like we are

> adding layovers to something that would be much faster if it

> was a direct flight. 

Seems like it, doesn't it? 🙂 That was exactly my expectation. Turns out in practice to not be a big deal. We really punished the AF with cases in its early days, and we've had several high-performance applications successfully written since then. Turns out that a) messaging is really fast on its own, b) most actors that need high performance hand-off are already adjacent in the tree structure and c) most up-and-over messaging needs some amount of coordination anyway and can't be short circuited by direct communication.

And if you do hit a performance bottleneck caused by the messaging overhead -- which I have heard of happening only once -- then you can share the enqueuer reference to allow specific actors to talk directly. That door is still open.

> Unfortunately I would have to spend a few years rewriting my applications in AF to find out.

I do not advocate that! 🙂 But as you are redesigning parts of your system, I do encourage you to keep the tree structure in mind and consider using it in your own architectures.

[jlokanis]

---

@AristosQueue (NI) wrote:

I do encourage you to keep the tree structure in mind and consider using it in your own architectures.

---

I do that already, except I have layers of trees with lots of siblings essentially.  So my trees are more like bonsais.  

[drjdpowell · Accepted solution]

---

@PrimaryKey wrote:

1. Obtain an actor enqueuer reference by name from an actor system.

5. Be able to provide global settings for an actor system.

All of these are natural consequences and would follow very closely to what is possible with any other (non-LV) actor framework out there ex. Akka. In Akka a local address to an actor can be /MyBaseActor/MyActor1 and remote address can be akka.tcp://MyActorSystem@localhost:9001/MyBaseActor/MyActor1 and these two would work exactly the same way. 

---

There is a concept in the “Actor Model” called “locality”, which appeals to the physicist in me as the same concept is in the physics of the Universe, but it might be better understood in programming as the concept of “privacy” (or “scope”).  Akka breaks this principle with its "Actor Paths”.   I don’t mind breaking rules, but one must always understand why the rule exists.

 It is the same issue as the difference between “named” and “unnamed” Queues in LabVIEW.  If I create an unnamed Queue, that Queue is private to me.  Noone else can access it, unless I choose explicitly to pass the Queue reference to them, and I can do stuff with the Queue and then pass it.  

With a “named” Queue, then the instant I create it others may already be acting on it (this “instant action-at-a-distance” is the “non locality” that does not exist in physics).  I have no control over the “privacy” of this Queue.  I don’t know who is accessing it.  

It’s a problem with an globally-accessible thing, including “action engines” and “IP addresses”.  The problem is minor if the number of components that could access something is only a few, because a single human programmer can easily understand all interactions of a few things at the same time.  But if there are many things (and “many” could be as few as five) then one can lose the ability to comprehend the entire program at once.  In this case, the ability to see that things are “private”, or are accessible only to a clear and limited set of things, is very valuable.

So when you say, basically, “wouldn’t it be great if every actor was globally accessible by any other?” I say “no”.  I’m not of AQ’s extreme where I don’t want you send messages “across the tree”, but I do want that ability to be explicitly set up (by passing address in messages).

Debugging is an exception, though.   As long as the debugger is an “information black hole” to the application, it doesn’t matter if it can globally access information about every actor.   The “Actor Manager” in Messenger Library is an example of this.

[AristosQueue (NI)]

I agree with Powell's points.

[PrimaryKey]

Good discussion. Thanks for all the replies.

[PrimaryKey]

How about this idea. All actors are registered with the arbiter and are fundamentally accessible but only based on their own private permissions ex. An actor is debugable locally, but not available for addressing. Another actor is addressable over the network but only to his network friend, who is also defined by path. There are many ways to specify the locality in a system like this. You would however be in a situation where the basic implementation is generic and public and you optimize by limiting it. This would be safe when critically important + easy. When I have an option I always go for generic and easy with options for limiting and optimizing instead the other way around. When it comes to teaching AF I gave the class 3+ times and every time it's difficult for attendees, even when they are advanced. This would point to AF being rather difficult. I personally think it could be much simplified by "breaking" the rules. Notice that Python is very simple, where everything is public. It became a huge success. People don't want to be limited. They want to make things work as fast as possible. I also think that DQMH and other frameworks should not exist, simply because AF is made to have everything, be easy, fast and best out of the box. They need to exist only because AF is lacking. I would argue we need to fix that - basic actors for simple tasks like logging and daq, template actors, tools, debugging, graphs, monitors, performance measurements, schedulers, deployers etc. Of course it's not my decision in the end. 

[Taggart]

---

@PrimaryKey wrote:

... I also think that DQMH and other frameworks should not exist, simply because AF is made to have everything, be easy, fast and best out of the box. They need to exist only because AF is lacking...

---

Simply having a really good hammer does not turn everything into a nail...  They exist because they fill different needs.  Good luck trying to write one framework that will work best in absolutely every situation...