Debugging Actors

[LVB]

I was wondering how everyone is debugging their Actor framework projects.  I have been using the DebugConsole from the Angry Eagles project.

• Are there any tools in development?
• To prevent queue overflow, is there a way to find the number of elements in an Actor's queue?

Looking forward to your responses...

[onnodb]

This is, I think, a very interesting question — one that apparently pops up all the time, as AQ seems to be implying With the great power of the AF come interesting bugs, which can be tricky to debug (e.g., race conditions).

Some techniques I've been using on and off:

• The Desktop Trace Execution Toolkit is invaluable for debugging the control flow in more sophisticated applications. The UI is klunky, but for most cases, it's a great help.
• Alternatively, you can roll your own logging system, like you've been doing with the AE DebugConsole.
• Although digging directly into Actor Cores is not really possible, given the re-entrant nature of the Actors, putting breakpoints inside "message methods" (i.e., the Actor methods that get called by Messages' Do VIs) does work.
• One thing that has been bugging me, is aborting "hanging" Actors — Actors that haven't shutdown properly. One way is to close the project you're working in, at which point LabVIEW will ask you if you want to abort any running VIs; needless to say, that's not a very efficient method.
  At the LavaG forums, there has been a post about some sort of "Actor Manager", based on the "LabVIEW Task Manager" from the same forum thread. I'm still curious to try and develop something similar myself.

Looking forward to hearing other people's ideas!

[Olivier-JOURDAN]

onnodb wrote:

• Although digging directly into Actor Cores is not really possible, given the re-entrant nature of the Actors, putting breakpoints inside "message methods" (i.e., the Actor methods that get called by Messages' Do VIs) does work.

Correct me if I'm wrong but you can add code to open Actor Cores ui at runtime in order to access the diagram currently running. You should had close FP property if you want to avoid to manualy close clone VI.

[Daklu]

onnodb wrote:

• The Desktop Trace Execution Toolkit is invaluable for debugging the control flow in more sophisticated applications. The UI is klunky, but for most cases, it's a great help.

I like the DETT for certain things, but many times the pain of dealing with the UI isn't worth it to me.  For Windows users, Damien posted a small vi that sends messages to the Windows debug output.  That, combined with DebugView, makes for a quick and easy logger to watch sequences of events.  (And it's free!)  I don't have the link but I think it is on the Nuggets thread in the forum.

[drjdpowell]

I experimented with an "Actor Manager" debug tool for the Actor Framework 2012beta in the conversation "The Actor Framework now ships with LabVIEW" in the LabVIEW Beta forum.  A problem is that one can't really do tools like this without customizing the Actor Framework itself to collect the information.

Another tool one could make is a tool to display the messaging relationships (who sends messages to who) with the option of "tapping" in to observe the message stream.  This is doable, but again requires hooking in to the framework itself.

-- James

[AristosQueue (NI)]

If you use the DETT with AF, please take a moment to post (start a new thread) any upgrades or feature enhancements you'd like to see in that tool. In theory (as usual, no promises for future functionality), there will be development work done for the 2013 release cycle to upgrade that tool again (it's been in bug fix/maintenance mode the last couple releases), and I might be able to shoehorn some AF specific features into that project. Maybe. Did I mention no promises? Yes? Good.

For the LV 2012 release, we've added a new Boolean terminal to Launch Actor.vi to open the Actor Core front panel of the launched clone (and close again when the clone finishes running), which makes it a lot easier to hit Abort on all the clones. The only drawback is you have to search-and-find all calls to Launch Actor.vi and wire True to that terminal. Or, I suppose, you could edit the VI temporarily to change the default value of that terminal. I try not to make edits like that -- I might accidentally submit them to the trunk along with some real changes to the framework. Oops. 🙂

No, there's no way exposed to investigate the queue elements. I did that deliberately -- the Get Queue Status primitive is one of the sources of weird race conditions in LV code. Anything I built to let you investigate the queue while debugging could be used by an inexperienced user in actual code, and that's exactly the sort of blind alley I'm trying to keep people from wandering into. However, as I'm typing this, I just had an idea... there is one primitive in LabVIEW that is broken *unless* it is used on the diagram of a custom probe. That's the "To Probe String" primitive (and it's broken on any other VI because it circumvents the encapsulation protections of LV classes). I could probably add custom probes for the queue classes. Hadn't really thought of that... not sure why... You could do the same for your copy of the Actor Framework. Sometimes, I feel like only part of my brain is "in cache" at any given time. 🙂

> A problem is that one can't really do tools like this without customizing the Actor Framework itself to collect the information.

Yes. This is a problem. I'm still wrestling with this myself. I think the only good solution in the near term is going to end up being adding code to the AF that is inside a Conditional Disable Structure and you just have to know to set the conditional tokens on your project to TRUE in order to enable the debug code. It isn't the most elegant solution, but anything else seems to require elevating the AF from a library that ships with LabVIEW to a fundamental part of the LV language, and I'm loathe to do that -- I hate claiming that a library written by NI is somehow more priviledged than a library written by any other user, and I try to make features that are available to NI VIs available to all users as much as possible. The only way I'd do this is if we decided that these Actors were going to be truly first class elements of LabVIEW, the same way that classes and VIs are first class citizens. But that would require a lot more confidence that this is the One True Way and not merely One Very Good Way Among A Few Good Ways In A Sea Of Bad Ways So We Want To Promote This One. 🙂

The list of tools that I'd like to have (many of which I have no idea how to build, even with full access to LV's C++ code at my disposal):

1. A way of single stepping into a "Send XYZ Message.vi" that, instead of going into that VI somehow sets a magic breakpoint such that when that particular message is dequeued on some far away actor, the VI pauses right before the Do.vi call.
2. A breakpoint that I can set on one actor that, when tripped, pauses *all* the other VIs that are running in parallel so that while I'm stopped at the breakpoint, messages don't keep piling up in the queue, events don't timeout, etc. And then, of course, the Unpause Everyone button to go with it.
3. A live runtime graphical view of the Actor Tree. Every call to Launch Actor.vi would look at the caller queue passed in and the actor queue passed out and update the tree. Every time an actor fired its Last Ack, it would disappear from the tree. This one I know we could build, but, again, requires instrumenting Launch Actor.vi, which I wish was unnecessary.
4. In that graphical view, the ability to click on any actor and pop open its front panel. Ah, bliss.
5. A system for opening the front panels of Actor Core VIs in a reasonable arrangement on screen and -- even better -- a viewer that takes all the inheritance overrides of Actor Core for a single actor and puts all of their block diagrams on a single block diagram, with all the loops in parallel, and lets me watch the execution highlighting for the whole kit and kaboodle.
6. Heck, I'll settle for a more limited version of #4: just open all the panels and block diagrams for one actor in one place on the screen and all the panels/diagrams for another actor on a different part of the screen, arranged in something like the Actor Tree.
7. An Abort All button on the Project window. This one I know how I want it to work, but I would have to learn a lot more about how the project window in LV works.
8. A way of getting VI references to running clones that didn't make the LV architects wince. You can use Open VI Reference and wire a string to it to open a VI by name. Apparently, no one but me realized that you can give it the name of a clone VI (with the colon+number after the file name) and get a reference to the clone VIs. Turns out, opening multiple references to a clone isn't exactly stable, and really cannot be made stable without redefining what a clone VI even means, so that magic trick may disappear in the future versions of LV. But I want something just like it, but stable, in the future. Then I could write several of the tools described above. Or get customers to do it for me and then use those tools. Even better!
9. A flight simulator version of execution highlighting where I'm in an airplane doing strafing runs down wires and barrel rolls to parallel wires, bombing nodes as they execute. No, this (probably) wouldn't help me debug Actor Framework at all, but it would be cool, and I've thought about building this for many years, just for the heck of it. "Roger, command, I see the data on the wire now. I'm in pursuit..." Definitely would add fun to the debugging experience.

In the words of one programmer friend of mine: "It's software. Anything's possible. How much are you paying me?" 🙂

[onnodb]

Olivier Jourdan wrote:

onnodb wrote:

• Although digging directly into Actor Cores is not really possible, given the re-entrant nature of the Actors, putting breakpoints inside "message methods" (i.e., the Actor methods that get called by Messages' Do VIs) does work.

Correct me if I'm wrong but you can add code to open Actor Cores ui at runtime in order to access the diagram currently running. You should had close FP property if you want to avoid to manualy close clone VI.

Ah, true; thanks for correcting me! I should try that...

[SteveChandler]

AristosQueue wrote:

     7. An Abort All button on the Project window. This one I know how I want it to work, but I would have to learn a lot more about          how the project window in LV works.

I know you already voted for it but I wanted to point out to others that there is an idea on the exchange for this.

http://forums.ni.com/t5/LabVIEW-Idea-Exchange/Abort-All-VIs-in-Project/idi-p/1818849

Also there is another one which is similar.

http://forums.ni.com/t5/LabVIEW-Idea-Exchange/Abort-all-VIs-Button-in-Taskbar-Modal-Windows-Problem/...

[drjdpowell]

The Actor Manager I made does 3 and 4.  2 and 6 are doable as extensions.  I use the open reference to clones by name trick to make things work, so I'm sad to hear this isn't stable.  What problems might occur in using this?

-- James

[AristosQueue (NI)]

drjdpowell wrote:

The Actor Manager I made does 3 and 4.  2 and 6 are doable as extensions.  I use the open reference to clones by name trick to make things work, so I'm sad to hear this isn't stable.  What problems might occur in using this?

-- James

The clones were never intended to have additional references. If you were to release the reference to the real VI while still holding references to the clones, such that the real VI feels free to unload from memory, it will take the clones with it, and leave stale pointers in the secondary references that are opened. LabVIEW fall down and go boom. It is likely ok in the AF situation because we never give the real VI permission to leave memory while any actor is still in play, but I'm not confident of that, and, honestly, the architects who created clone VIs were so disturbed by my showing them that the open-by-name trick works, I can't rule out there being some other effect, so I'm loathe to rest a tool of AF on that trick. Fixing that crash would require significant refactoring of the clone VIs and almost certainly put some performance penalties on some parts of the operation of clones (can you return a clone to the clone pool when its first reference is released but there are secondary references still open?). Moreover, I don't remember how you figured out which clone numbers to try in your tool, since there's no way to get a list of the clone IDs that are currently in use, so I really had shied away from investigating further... I was assuming that you were instrumenting Actor.vi, which, as I've mentioned, I've been avoiding thus far (though I'm coming to the conclusion that is what we're going to have to do).

If we do instrument Actor.vi, we should be able to use the "This VI" primitive on the block diagram of that VI to get the first clone VI refnum, which means we would not be creating any additional clone references, and then register that with a central debugger.  So, when I said above that I wasn't sure how to make this work, that was mostly because I couldn't do it without instrumenting Actor.vi.