High Performance library crash in OMP?

[altenbach]

Well, here's a new one. I played a bit with this code, except that I replaced the code in the first code frame with a parallel FOR loop as posted here (configurd for 32 parallel instances)

In summary, we have a parallel matrix multiply and a parallel FOR loop with 32 instances, if that matters.

I placed a chart on the three timing outputs and did a "continuous run" for a quick a dirty test. It runs fine, but after about 100 (?) repeats LabVIEW crashes and we get a console windows as follows. I assume OMP means OpenMP or something. Obviously we have a serious resource leak somewhere under these conditions. Reproduced twice.

 

The crash no longer seems happens if I disable the loop parallelism. Curiously after re-enabling the parallelism it also no longer crashed but I still got a automatic error report at the end (60041bd0-c40f-4b9c-a945-c50fdb0347b7).

I'll investigate some more....

(This is on a dual E5-2687W (32 virtual cores) with 32GB of RAM)

[GatorChomp]

Hi altenbach,

I tried to recreate the issue on my end and did not get any kind of error, though I only have one processor with 8 virtual cores as opposed to your two with 32 total virtual cores. I tried the snippet below both with the while loop and without the while loop using a continuous run, and both times ran it for a few minutes without problem. Is the below snippet the same as what you are using (minus the while loop of course)?

I'm curious if anyone else can recreate this problem on a different system. 
Does the error still happen regardless of how many instances of the parallel FOR loop?

[mcduff]

Look at the link

http://forums.ni.com/t5/LabVIEW/optimization-nested-loops-for-speed/m-p/3225565#M937392

I believe he wants to parallelize the code with the High Performance Library as in the link.

It was my understanding that the High Performance Library automatically parallelizes the code across the cores, I wonder ifthe error is related to try to parallelize something even further.

Cheers,

Andrew

[altenbach]

I have the code at work and will attach it later. You have one extra transpose in the first frame, so it is not quite the same as I remember.

As I said, after i disabled loop parallelism once more, then re-enabled it again, it no longer seemed to crashed the same way. Maybe it was just in a weird state? Maybe it tried to use more threads than it could?

Can you comment on the OMP error console popup? I've never seend that before! My guess is that it is caused by the high performance subVI. Are these documented somewhere?

[altenbach]

Yes, it still crashes with the attached VI (no guarantees that it its mathematically correct ;))

This time, after a fresh windows update and reboot, it took definitely longer (several hundred interations), but it crashed the same way.

As you can see, all CPU cores were pretty busy at once when I pressed continuous run. The time axis is 60 seconds so I am guessing ~10s of running before it crashed.

The LabVIEW 2015 icon in the task bar just goes to that console window. No other LabVIEW windows (GSW, etc.) visible. After closing the console window and restarting LabVIEW, there is no offer to report any crash to NI.

 

[GatorChomp]

Hi altenbach,

That code still runs fine on my end, and I ran it continuously for a few minutes with no issues.

After looking into that OMP error and the crash log from your original post, I think you might be dead on with the thread creation. That error seems to occur when a program tries to create a new thread while the maximum number already exist, and the crash log from before contained an access violation error. The problem is I'm not sure what could be causing it. I haven't ever seen a crash which pulls up an OMP window like that. 

Have you tried isolating the parallel FOR loop into a separate VI? I'm curious if it's possible that the High Performance Library matrix multiplication sub VI is somehow trying to create an extra thread before the ones from the parallel FOR loop have been released. 

Another thing to check: What is the LabVIEW memory usage during the run of that VI? If it's 32-bit LabVIEW it could be the program is trying to create a thread when it doesn't have enough memory available to allocate to it. I highly doubt that's the case, but it's worth checking.

[altenbach]

Memory usage is constant and not that dramatic. (I have to hurry to get the screenshot before it crashes :D)

 

I'll try the subVI suggestion later ....

[altenbach]

---

@altenbach wrote:

I'll try the subVI suggestion later ....

---

Putting the parallel loop into a subVI causes exactly the same crash (after ~200 iterations)

[altenbach]

---

@GatorChomp wrote:

... though I only have one processor with 8 virtual cores as opposed to your two with 32 total virtual cores.

---

Now is the perfect time to go to your supervisor and ask for 32 core machine for testing! :o:D

(There is even a 18 core Xeon E7-8890V3 that allows up to 8 such CPUs per system (144 real cores and 288 virtual cores!). Probably less than $100k for a maxed out rig 🐵

[GatorChomp]

Hi altenbach,

Just to make sure I understand, that specific parallel FOR loop encounters the error even if it's the only thing you run?

Do all other parallel FOR loops work without problem? You can just try the example "Parallel For Loop Reduction.vi" from the LabVIEW Example Finder if you don't have other code readily available. If other parallel FOR loops work without trouble it sounds like there's something about the A x B sub VI which is causing the crash.  

And I wish I could grab a system like that!