Configure the number RT Threads to match the number of CPU's. Good or bad?

[sparkymark567]

Each execution system of the LabVIEW Real Time engine has 4 RT threads per CPU at each priority level.
(Except for Time Critical Priority which has 1 Thread per CPU.)

Application threads, i.e. parallel processing paths, are not directly coupled to RTOS threads!
LabVIEW code is divided into tasks that are placed into run queues.
Each execution system has a run queue for each priority level.
In many cases, it is best to leave the VI properties for 'Execution System' and 'Priority' alone, so that all tasks are run from the same queue.

Each RTOS thread executes tasks from the run queue and is allowed to run for 10ms, before the RTOS switches threads.
After the RTOS has switched threads, the 2nd thread then executes tasks from the same run queue.
Then the 3rd thread and the 4th.

I have noticed that when a thread gets pre-empted, the executing task is not got returned to the run queue.
The task remains in a partially executed state, until the RTOS reactivates the thread.
(This behavior was observed using cRIO-9074 and the Real Time Trace Viewer)

The result is that tasks of equal priority do not get executed an equal number of times in a round robin schedule.

Example:
Thread 0 executes tasks 1, 2, 3 ,4, 5, 6, 7 .... and task 8 is pre-empted during execution because 10ms has elapsed.
Thread 1 executes tasks 1, 2, 3, 4, 5, 6, 7 .... and task 1 is pre-empted during execution because 10ms has elapsed.
Thread 2 executes tasks 2, 3, 4, 5, 6, 7, 2 .... and task 3 is pre-empted during execution because 10ms has elapsed.
Thread 3 executes tasks 4, 5, 6, 7, 2, 4, 5 .... and task 6 is pre-empted during execution because 10ms has elapsed.
Thread 1 executes tasks 8, 2, 4, 5, 7, 8, 2 .... and task 4 is pre-empted during execution because 10ms has elapsed.

...

The system causes an unexpected execution sequence as well as task jitter.
It seems that the design/configuration of the RT Engine is flawed?
One possible solution, is to configure each execution system (priority) within the run-time engine with 1 thread per CPU.

But why are there 4 threads in the first place?
If i configure the RT Engine with 1 thread (per CPU) per execution system (priority), would it have unintended consequences?

[billko]

This is not a technical answer, but I'd say, let LV and its thirty-plus years of experience managing its own threads continue to do so.  It is very good at managing itself.

Edit: I re-read your post and it's not working for you in your case.  I'm not sure if it would help or not, but you can go ahead and try and then see if it helps.  You could always go back to a "pre-change" revision if it doesn't work out.

[sparkymark567]

I spent a long time investigating how the RT engine actually works and a lot of the information out there is based on incorrect knowledge and/or assumption. I have been a LabVIEW developer for more than 10 years....and it turns out, some things I thought I knew were wrong.

For example: one developer stated that subroutines are executed atomically. i.e. the RTOS will not switch tasks while executing a subroutine. But this is incorrect, because a subroutine only prevents the current RTOS thread from switching tasks. Application code (tasks) can be executed by a number of different RTOS threads, since each run queue is accessed by multiple threads (by default 4 threads per CPU). Each time an application thread/task performs a wait, or is blocked, the task is returned to the run queue. The next time the task executes, there is a high chance it will be executed on a different thread!

If a task gets picked off a run queue by an RTOS thread that has already executed for 9.9ms, the task will execute for just 0.1ms before it is pre-empted and blocked for potentially for 100's of ms. Meanwhile, other tasks of equal priority in the same execution system can execute multiple times on the other threads. The behavior is not what you expect from a real time system.

If a task executes for more than 50ms (total execution time, not relative time) the task is returned to the run queue so that the execution time of the thread can be shared with other tasks. It's seems strange to employ this mechanism, when each execution system (priority) has 4 threads. If one thread becomes occupied with a long running task, there are still 3 more threads to execute other tasks. Or to look this it the other way, the RT Engine already has a mechanism to prevent one task from hogging the available execution time of a thread, so why have 4 threads (per CPU) each performing the same function.

(Note: 10ms is the thread execution time for cRIO-9074, for cRIO-9067 it's 5ms).

[billko]

---

@sparkymark567 wrote:

I spent a long time investigating how the RT engine actually works and a lot of the information out there is based on incorrect knowledge and/or assumption. I have been a LabVIEW developer for more than 10 years....and it turns out, some things I thought I knew were wrong.

For example: one developer stated that subroutines are executed atomically. i.e. the RTOS will not switch tasks while executing a subroutine. But this is incorrect, because a subroutine only prevents the current RTOS thread from switching tasks. Application code (tasks) can be executed by a number of different RTOS threads, since each run queue is accessed by multiple threads (by default 4 threads per CPU). Each time an application thread/task performs a wait, or is blocked, the task is returned to the run queue. The next time the task executes, there is a high chance it will be executed on a different thread!

If a task gets picked off a run queue by an RTOS thread that has already executed for 9.9ms, the task will execute for just 0.1ms before it is pre-empted and blocked for potentially for 100's of ms. Meanwhile, other tasks of equal priority in the same execution system can execute multiple times on the other threads. The behavior is not what you expect from a real time system.

If a task executes for more than 50ms (total execution time, not relative time) the task is returned to the run queue so that the execution time of the thread can be shared with other tasks. It's seems strange to employ this mechanism, when each execution system (priority) has 4 threads. If one thread becomes occupied with a long running task, there are still 3 more threads to execute other tasks. Or to look this it the other way, the RT Engine already has a mechanism to prevent one task from hogging the available execution time of a thread, so why have 4 threads (per CPU) each performing the same function.

(Note: 10ms is the thread execution time for cRIO-9074, for cRIO-9067 it's 5ms).

---

I've also been a LV developer for about ten years.  And I also am constantly revising what I actually do know vs what I thought I knew.