Consumer loop gets slower and slower in a long duration monitoring process

[abvenk]

Hello,

I have a very standard QMH architecture, with a UI "producer" loop which handles front panel controls and sends messages to the consumer loops for the appropriate action.

One of the consumer loops has a state machine with standard states like Init, Read Queue, Exit and some custom action states.

Now, one process in particular involves continuous monitoring at a specified polling rate (currently set to 1s). The user is allowed to stop this process by clicking a Stop button. To enable this, I am going back and forth between the state where monitoring code is executed and the Read Queue state in order to stay responsive to a stop message. 

This works perfectly at the beginning for quite a while. But then I noticed the UI responsiveness slow down and the program seemed to take up a lot of memory and slow down.

Am I doing something fundamentally wrong here? How do I stay responsive to UI messages while monitoring a long process at periodic intervals?

[aeastet]

This could be a lot of different causes. You could have a memory leak. You could be building an array that is just getting too big. It is hard to say without seeing your code. One thing that you can look for is in task manager after a few hours do you see the memory for this app go up. If so it is a memory issue.

[crossrulz]

Just from the title, my mind instantly went to "ever-growing array in a loop". Beyond that educated guess, we would need to see actual code to give any type of diagnosis on what could be slowing your application down.

[abvenk]

There is no ever-growing array in any loop. 

I'm trying some troubleshooting now, like seeing the loop rate to find out when it changes.

The loop that has this back-and-forth also uses Python nodes. I know the Python nodes can be blocking, but I made sure to write all my Python functions to be non-blocking, so any single Python node shouldn't get stuck waiting for the Python code to finish. 

Sorry, it's difficult to post the code here because it's confidential but re-creating it would take too long. But there is no growing array. That would be a very quick find and fix. 😊

[abvenk]

---

@aeastet wrote:

This could be a lot of different causes. You could have a memory leak. You could be building an array that is just getting too big. It is hard to say without seeing your code. One thing that you can look for is in task manager after a few hours do you see the memory for this app go up. If so it is a memory issue.

---

I don't see the memory usage go up significantly. But I will pay close attention again.

[JÞB]

---

@crossrulz wrote:

Just from the title, my mind instantly went to "ever-growing array in a loop". Beyond that educated guess, we would need to see actual code to give any type of diagnosis on what could be slowing your application down.

---

Or, an ever growing Queue!  Probe the queue for number of elements.  Don't forget the event queue! Which Event cases defer FP?  Which do not?  The Event Inspector window will show a lot of common mistakes. 

[abvenk]

No ever-growing queue either! I'm explicitly monitoring queue sizes. 

And I'm checking the loop time for this loop. In idle state the queue timeout of 250ms applies. During the monitoring process, it is 20-25ms.

[Mads]

This is normally something that can be spotted easily if we have a look at the code...If it contains proproetary stuff, try stripping that away and reproduce the issue, if it reproduces you can share that...if not it might indicate where the issue is already...

[abvenk]

---

@Mads wrote:

This is normally something that can be spotted easily if we have a look at the code...If it contains proproetary stuff, try stripping that away and reproduce the issue, if it reproduces you can share that...if not it might indicate where the issue is already...

---

Okay, I've stripped away everything other than the loop in question and attached the code snippet. Obviously, I cannot make this code snippet work because it interacts with other loops. All the missing VIs contain Python nodes calling specific Python functions. There is no blocking due to the Python nodes.

But the heart of the problem, as mentioned earlier, is the slow down during the monitoring process which involves switching between the states "Idle" and "Monitor Establish Count Time" which occurs according to the polling interval (currently 1 second).

Looking forward to your inputs!

[Mads]

Nothing obvious in the framework itself, perhaps something is building up in any of the many missing subVIs...Have you tried logging which states are executed and their execution time...Just streaming it to an open text file e.g., to see if any particular parts are slowing down and at which rate? If any of the underlying functions are building arrays, reopening resources, files etc? There is a save to file function that might indicate that you are budiling the content of that e.g...