ACBR Clones Execute Serially - Sometimes

[Mr._Jim]

Alright, I'm observing something that's not making sense, and maybe someone has seen this before. I'm hoping I've just missed something trivial.

• I've got a preallocated pool of clone VIs
• I'm using the call and collect method
• I'm using 0x100 and 0x40 for the "options" flag
• I've got my clone VIs set as shared clones.
• I'm reusing the single VI ref so as not to enqueue a request to the root loop. (Except the first time.)

We run IO traces to troubleshoot an unrelated issue, and we log IO trace messages as well, when each clone starts and when each clone ends execution. (Kind of a weak version of the DETT)

In an hours-long run, I see the clones execute serially in the IO trace. One will run, exit, and the next will run and exit, as if they're just running in a for loop. There are non-reentrant dependencies inside each clone VI, but in the past I've seen the clone VIs run asynchronously as expected.

Why would these VI clones execute serially sometimes? Has anyone else seen this occur?

I've used this method a bunch of times and never seen this behavior.

This is on LabVIEW 2012, (we will eventually upgrade, but it's not an option right now)

Thanks,

Jim

[Bob_Schor]

This looks a little strange to me, and I would like to examine the cases that aren't shown on your picture.  Can you please attach the VI, or a LabVIEW Snippet that will let us examine the entire VI?

It seems to me that if you have a reference to a clone and you call it via that reference, you are, in fact, calling the same clone over and over, so it has to be called serially.  Maybe I'm mis-understanding something (and without the full VI I can't easily "experiment" and try to figure this out), but maybe your code is doing what you told it to do ...

Bob Schor

[Mr._Jim]

Hi Bob, thanks for replying.

> Can you please attach the VI, or a LabVIEW Snippet that will let us examine the entire VI?

Sadly, I cannot post the code itself, due to its proprietary nature. I am using the methodology illustrated in the example, AsynchronousCallAndCollectUsingOption0x40.vi

(Except that in the version of this example I'm looking at, the callee VI is set as a preallocated clone, which has never seemed necessary to me.)

Please excuse the less than optimal data structures... I didn't come up with those.

The green VI on the left is the subVI I illustrated in a prior post.

I can't show you the clone. However, if I strip down my code to run without any hardware, I see it run asynchronously with varying clone order (as expected). When I put my code on a test stand and run it for hours, it runs serially.

I'm not really expecting an answer to jump out... I mostly just wondered if anyone else has ever experienced anything similar.

[Bob_Schor]

This got me thinking about a project I did four years ago that involved collecting data from 24 stations, each with a pair of DAQ hardware (a VISA device and a video camera).  We created a single reentrant VI for the VISA device and the Camera, but when we spun them up (at the beginning of the program using a For loop that ran 24 times), we used Option 0x100, Call and Collect, but not 0x140.  Instead, we "collect" the 24 refnums for each Clone, save them in an Action Engine, and use them when the Clone quits.  We definitely have all 24 stations running in parallel, capturing videos at 30 frames/sec ...

Bob Schor

[Mr._Jim]

Hmm... The implementation I'm replacing with this would open a series of clone references, pretty much as you described. They were using the ACBR node to call each clone from an array of references. Then they'd close the clone references. The loading and closing of references was adding a lot of overhead, so I modified it to use a single ref. It works very, very efficiently, except for the bizarre and inconsistent serial execution.

I suppose another option would be to store an array of refs in an action engine, but I really don't want to go back to the array of refs if I can help it.

[Bob_Schor]

Never thought about overhead in opening/closing references.  Maybe I'll play around with this a bit, see what I can figure out.  It will have to be later, though -- I'm off to Mens Glee Club rehearsal ...

BS

[Mr._Jim]

Alright, well thanks for the $0.02 Bob. I miss being involved in more musical activities, myself.

[Sam_Sharp]

---

@Mr._Jim wrote:

• I'm reusing the single VI ref so as not to enqueue a request to the root loop. (Except the first time.)

---

My understanding of this is different - I was under the impression that it's only a request to the root loop if you specify the VI by Path (causing a disk read), rather than by Name.

Are you sure the problem isn't due to a deadlock inside the VIs themselves? Perhaps you have some dependencies on the UI thread which is blocking the execution or you're waiting for hardware to be available?

One thing I find helpful with asynchronous calls is to temporarily set the VIs to show front panel when called - if all is well then you'll have all of your front panels open. If there's a deadlock inside the VI then you'll see them all open, but close one-by-one. If the VI itself is actually being called sequentially then the panels will appear one at a time.

[Mr._Jim]

Hi Sam,

Thanks for the response.

Ah, we're back to the root loop stuff!

> My understanding of this is different - I was under the impression that it's only a request to the root loop if you specify the VI by Path (causing a disk read), rather than by Name.

According to various sources, other various sources, which elaborate a bit further than the LabVIEW help, using a string containing the VI name does not involve the root loop (what you're saying is correct). Using the string name with a type specifier, however, does involve the root loop. I wasn't sure I believed it myself until I tested it. The test I used was launching a single button dialog in parallel to ACBR calls, and it proved very effective in illustrating the symptoms. Now, because the ACBR node relies on a defined type, there doesn't seem to be a way around this.

Anyway, all that aside, the above code gets around the issue by only bothering the root loop once, and then caching the reference. (Here's another thread referring to the same technique)  Based on my benchmarks, it's among the most efficient ways to repeatedly launch worker threads.

> Are you sure the problem isn't due to a deadlock inside the VIs themselves?

> If there's a deadlock inside the VI then you'll see them all open, but close one-by-one. If the VI itself is actually being called sequentially then the panels will appear one at a time.

I'm certainly not ruling out internal "deadlock," but it's weird because in the past, I've seen runs where the VIs start together, fight over resources, and then finish executing as the resources are freed up, much as you've described. That's different than what I'm seeing now, where they basically queue up and run one at a time. I double and triple checked all the settings, and everything seems in order for them to run asynchronously.

> One thing I find helpful with asynchronous calls is to temporarily set the VIs to show front panel when called

Yeah, I've done that, too. I have a debugging subVI that I can place in my clones such that when they're called the first time, their panels open. This gets around the annoying behavior of the panels continuously opening if you just use the VI property setting.

What's weird is when I test it and observe it, it runs correctly every time. When I run it on the test stand as part of a TestStand sequence, the worker thread VIs appear to queue up and run serially as if in a for loop.

(No, I did not come up with this, since TestStand already does multithreading well. I am trying to get away from calling elaborate LabVIEW code from TestStand, but a lot of existing stands already use this technique and I am supporting that code)

Forgive the missive - I tried to keep my word count down. 

Well, if nothing else, it certainly makes for interesting discussion.

[Mr._Jim · Accepted solution]

Alright, this is weird.

We ran the code on the stand this morning and everything ran asynchronously. The only thing I changed was the following:

Before:

Before, in each clone I was querying the clone name, simply to label when the clone did things in the IO trace log.

After:

Turns out when you use the single ref method with the ACBR node, the CloneName property doesn't work.

The CloneName property was returning a blank string. It does retun something valid under normal circumstances. Why it wasn't working for the earlier runs, I don't know.

So, instead, I used the following to label each clone in the IO trace log:

Once I switched to the "after" method, everything seemed to run asynchronously. Clones start up together, negotiate non-reentrant resources, and end at various times as expected.

It probably sounds crazy, but that's what I'm observing. Is it possible I inadvertantly did something else subtle that restored asynchronous behavior? Sure, but I don't know what it might be.

The symptoms to me look like I must have goofed and specified the wrong options flag, but I triple checked that flag, and didn't change it for the last run that ran properly.

This is likely one of those things... I'll never really know why it did what it did.