Networking and kernel workers

[Fotis Panagiotopoulos <f....@gmail.com>]

Hello,

As far as I know, the network must use the low-priority workers.
I can see a comment in arch/arm/src/stm32/stm32_eth.c line 88.
But I also remember reading somewhere that the high priority workers may
cause dead-locks.

But I think that the difference between the two workers is only the...
priority.
I guess that someone could easily swap their roles but adjusting the
priorities.

Which raises the question: what is a "high" priority?
The low-priority workers shall have a priority less than what?

For example, I have several tasks (that use the network) running at a
priority of 100.
For various (irrelevant) reasons I raised the LP workers to 105.
Is this correct, or can it cause problems?

In general, which tasks must run at a higher priority than the LP work?

*(I am working on an STM32F4, if it makes any difference)*

Re: Networking and kernel workers

[Gregory Nutt <sp...@gmail.com>]

You said one other thing that caught my attentioni:

> For example, I have several tasks (that use the network) running at a
> priority of 100.

In a properly turned real time system, you should never have two tasks 
with the same priority.  That breaks some of the fundamental assumptions 
that support deterministic response times.  If two tasks, A and B,  have 
the same priority, the response time of B will be scrambled, depending 
on if A is running or not and how close A is to releasing the CPU to 
allow B to run.

Strict priority scheduling is necessary for good real time performance.

Re: Networking and kernel workers

[Gregory Nutt <sp...@gmail.com>]

On 4/1/2023 12:07 PM, Fotis Panagiotopoulos wrote:
>>> Is there any limitation anywhere that dictates the relationship between
>>> these two priorities?
>> Since they run synchronously and generally don't overlap in time, I
>> would say that the answer is no
> I think this answers my question, thank you.
>
> So, It doesn't really matter what the priority of the workers is (OS-wise),
> as long as they do not interfere with other high-priority real-time tasks.
Maybe, assuming that the worker is not performing real time actions 
(which would be a bad idea).

Re: Networking and kernel workers

[Fotis Panagiotopoulos <f....@gmail.com>]

>> Is there any limitation anywhere that dictates the relationship between
>> these two priorities?
> Since they run synchronously and generally don't overlap in time, I
> would say that the answer is no

I think this answers my question, thank you.

So, It doesn't really matter what the priority of the workers is (OS-wise),
as long as they do not interfere with other high-priority real-time tasks.



On Sat, Apr 1, 2023 at 6:27 PM Gregory Nutt <sp...@gmail.com> wrote:

>
> > The network API (sockets etc) will be called by the user's task.
> > The network driver will run on the LP worker.
>
> The socket calls run at the user priority, but no not actually do the
> transfers.  The just set up to do the transfer then wait for the
> transfer to complete.
>
> The transfer itself is initiated by the driver when there are buffers
> and hardware resources available to do the transfer.  So the whole
> transfer runs on the LP work thread.  The user thread is not involved
> since it is just waiting all this time.
>
> Other network actions (like most IOCTLs) are performed entirely on the
> user thread.
>
> > Is there any limitation anywhere that dictates the relationship between
> > these two priorities?
> Since they run synchronously and generally don't overlap in time, I
> would say that the answer is no
> >> In a properly turned real time system, you should never have two tasks
> >> with the same priority.  That breaks some of the fundamental assumptions
> >> that support deterministic response times.  If two tasks, A and B,  have
> >> the same priority, the response time of B will be scrambled, depending
> >> on if A is running or not and how close A is to releasing the CPU to
> >> allow B to run.
> > I think this shouldn't be an issue. Tasks with actual real-time
> > requirements are at higher and distinct priorities.
> > Everything else is at (or around) 100 with a round-robin policy. There
> are
> > no dead-lines to meet there, so I don't
> > care about any jitter.
> >
> > Also there are two workers in the LP queue, not one.
> > So far I haven't noticed any availability issues, but something that
> looks
> > more like a dead-lock to me.
>
> Yes, this is only for tasks with real time requirements.  There should
> be an segregation of priorities into two groups:  All of the background
> tasks that just need to do things as time permits should be in the lower
> priority group.  All task that have deadlines to meet should be in a
> separate group of with higher priorities.
>
> That way, then non-real-time background tasks will not interfere with
> real time behavior.
>
> Linux does that:  All of the SCHED_OTHER tasks are lower priority than
> any of the SCHED_FIFO tasks.
>
>

Re: Networking and kernel workers

[Gregory Nutt <sp...@gmail.com>]

> The network API (sockets etc) will be called by the user's task.
> The network driver will run on the LP worker.

The socket calls run at the user priority, but no not actually do the 
transfers.  The just set up to do the transfer then wait for the 
transfer to complete.

The transfer itself is initiated by the driver when there are buffers 
and hardware resources available to do the transfer.  So the whole 
transfer runs on the LP work thread.  The user thread is not involved 
since it is just waiting all this time.

Other network actions (like most IOCTLs) are performed entirely on the 
user thread.

> Is there any limitation anywhere that dictates the relationship between
> these two priorities?
Since they run synchronously and generally don't overlap in time, I 
would say that the answer is no
>> In a properly turned real time system, you should never have two tasks
>> with the same priority.  That breaks some of the fundamental assumptions
>> that support deterministic response times.  If two tasks, A and B,  have
>> the same priority, the response time of B will be scrambled, depending
>> on if A is running or not and how close A is to releasing the CPU to
>> allow B to run.
> I think this shouldn't be an issue. Tasks with actual real-time
> requirements are at higher and distinct priorities.
> Everything else is at (or around) 100 with a round-robin policy. There are
> no dead-lines to meet there, so I don't
> care about any jitter.
>
> Also there are two workers in the LP queue, not one.
> So far I haven't noticed any availability issues, but something that looks
> more like a dead-lock to me.

Yes, this is only for tasks with real time requirements.  There should 
be an segregation of priorities into two groups:  All of the background 
tasks that just need to do things as time permits should be in the lower 
priority group.  All task that have deadlines to meet should be in a 
separate group of with higher priorities.

That way, then non-real-time background tasks will not interfere with 
real time behavior.

Linux does that:  All of the SCHED_OTHER tasks are lower priority than 
any of the SCHED_FIFO tasks.

Re: Networking and kernel workers

[Fotis Panagiotopoulos <f....@gmail.com>]

Thank you very much for this explanation.

> This is really outside of the scope of the OS.  There is no OS imposed
> rule about what priorities you should use for your applications or for
> the low priority work queue.  So I don't think there is a rule or a
> correct answer.  You just need to experiment and tune your system so
> that it behaves as you like.

I am not sure about this, that's why I am asking. Let me rephrase.

The network API (sockets etc) will be called by the user's task.
The network driver will run on the LP worker.

Is there any limitation anywhere that dictates the relationship between
these two priorities?
Either limitation in the (STM32) drivers, or in the TCP/IP stack itself?

> In a properly turned real time system, you should never have two tasks
> with the same priority.  That breaks some of the fundamental assumptions
> that support deterministic response times.  If two tasks, A and B,  have
> the same priority, the response time of B will be scrambled, depending
> on if A is running or not and how close A is to releasing the CPU to
> allow B to run.

I think this shouldn't be an issue. Tasks with actual real-time
requirements are at higher and distinct priorities.
Everything else is at (or around) 100 with a round-robin policy. There are
no dead-lines to meet there, so I don't
care about any jitter.

Also there are two workers in the LP queue, not one.
So far I haven't noticed any availability issues, but something that looks
more like a dead-lock to me.



On Sat, Apr 1, 2023 at 5:29 PM Gregory Nutt <sp...@gmail.com> wrote:

>
> > As far as I know, the network must use the low-priority workers.
> > I can see a comment in arch/arm/src/stm32/stm32_eth.c line 88.
> > But I also remember reading somewhere that the high priority workers may
> > cause dead-locks.
> >
> > But I think that the difference between the two workers is only the...
> > priority.
> > I guess that someone could easily swap their roles but adjusting the
> > priorities.
>
> The high priority work queue was intended only to serve as the back end
> for interrupt handlers:  If interrupt handling needs more extensive
> processing, then most of the interrupt processing can be off loaded to
> the high priority work queue.
>
> In order for the high priority work queue to be able to do that job:
>
> 1. It must be IDLE most of the time.  If it is busy processing a lot of
> nonsense, then back end interrupt processing will be deferred and
> probably cause bad behaviors for the system
>
> 2. It must be VERY high priority.  The default is 224.  I would use 255
> (maximum priority).  Nothing must prevent the high priority work queue
> from running and finishing interrupt processing.
>
> Unfortunately, the high priority work queue gets over-used for other
> things and often interferes with its primary purpose of servicing
> interrupts.
>
> The low priority just run a a priority similar to most task.  it
> defaults to a priority of 100.  It is intended as a convenience for when
> you just need to do a small something on a thread.  It is not intended
> for extended processing and things running on either queue should not
> block.
>
> Care should be used when using either work queue.  The are not real time
> friendly.  Things deferred to a busy work queue will happen at some
> random time after queuing and that random time can be quite large.
>
> I don't know if this is valid syntax, but the low priority work queue
> should have a priority range in the Kconfig like:
>
>      range 1 (SCHED_HPWORKPRIORITY-1)
>
> > Which raises the question: what is a "high" priority?
> > The low-priority workers shall have a priority less than what?
> The absolute value of a priority is irrelevant.  Higher priority threads
> run before lower priority threads.  The absolute numeric value of the
> priority does not affect when the task runs, only its relative priority
> to the priority of other threads.
> > For example, I have several tasks (that use the network) running at a
> > priority of 100.
> > For various (irrelevant) reasons I raised the LP workers to 105.
> > Is this correct, or can it cause problems?
> >
> > In general, which tasks must run at a higher priority than the LP work?
> >
> > *(I am working on an STM32F4, if it makes any difference)*
>
> This is really outside of the scope of the OS.  There is no OS imposed
> rule about what priorities you should use for your applications or for
> the low priority work queue.  So I don't think there is a rule or a
> correct answer.  You just need to experiment and tune your system so
> that it behaves as you like.
>
> A couple of other interesting thing about the lower priority work queues:
>
> - You can have multiple low priority work queues.  The are maintained in
> a thread pool and can service multiple lower priority work requests
> concurrently.  You might need multiple low priority work queues if there
> is a possibility that one may block or delay and constipate one work queue
>
> - The lower priority work queue is intended for use only within the OS.
> There is also a user space work queue that uses only standard POSIX
> application interfaces and so it more suitable for use by applications.
>
> >
>
>

Re: Networking and kernel workers

[Gregory Nutt <sp...@gmail.com>]

> As far as I know, the network must use the low-priority workers.
> I can see a comment in arch/arm/src/stm32/stm32_eth.c line 88.
> But I also remember reading somewhere that the high priority workers may
> cause dead-locks.
>
> But I think that the difference between the two workers is only the...
> priority.
> I guess that someone could easily swap their roles but adjusting the
> priorities.

The high priority work queue was intended only to serve as the back end 
for interrupt handlers:  If interrupt handling needs more extensive 
processing, then most of the interrupt processing can be off loaded to 
the high priority work queue.

In order for the high priority work queue to be able to do that job:

1. It must be IDLE most of the time.  If it is busy processing a lot of 
nonsense, then back end interrupt processing will be deferred and 
probably cause bad behaviors for the system

2. It must be VERY high priority.  The default is 224.  I would use 255 
(maximum priority).  Nothing must prevent the high priority work queue 
from running and finishing interrupt processing.

Unfortunately, the high priority work queue gets over-used for other 
things and often interferes with its primary purpose of servicing 
interrupts.

The low priority just run a a priority similar to most task.  it 
defaults to a priority of 100.  It is intended as a convenience for when 
you just need to do a small something on a thread.  It is not intended 
for extended processing and things running on either queue should not block.

Care should be used when using either work queue.  The are not real time 
friendly.  Things deferred to a busy work queue will happen at some 
random time after queuing and that random time can be quite large.

I don't know if this is valid syntax, but the low priority work queue 
should have a priority range in the Kconfig like:

     range 1 (SCHED_HPWORKPRIORITY-1)

> Which raises the question: what is a "high" priority?
> The low-priority workers shall have a priority less than what?
The absolute value of a priority is irrelevant.  Higher priority threads 
run before lower priority threads.  The absolute numeric value of the 
priority does not affect when the task runs, only its relative priority 
to the priority of other threads.
> For example, I have several tasks (that use the network) running at a
> priority of 100.
> For various (irrelevant) reasons I raised the LP workers to 105.
> Is this correct, or can it cause problems?
>
> In general, which tasks must run at a higher priority than the LP work?
>
> *(I am working on an STM32F4, if it makes any difference)*

This is really outside of the scope of the OS.  There is no OS imposed 
rule about what priorities you should use for your applications or for 
the low priority work queue.  So I don't think there is a rule or a 
correct answer.  You just need to experiment and tune your system so 
that it behaves as you like.

A couple of other interesting thing about the lower priority work queues:

- You can have multiple low priority work queues.  The are maintained in 
a thread pool and can service multiple lower priority work requests 
concurrently.  You might need multiple low priority work queues if there 
is a possibility that one may block or delay and constipate one work queue

- The lower priority work queue is intended for use only within the OS.  
There is also a user space work queue that uses only standard POSIX 
application interfaces and so it more suitable for use by applications.

>