You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nuttx.apache.org by "Matias N." <ma...@imap.cc> on 2021/01/26 22:26:48 UTC
limitation in SIGEV_THREAD?
Hi,
working with nimBLE I found that I had an issue when scheduling a callback to be made from within the signal handler for a timer, which was set with SIGEV_THREAD. The issue was that I was pushing to a POSIX queue from within the handler, and it was failing with BADFD. From debugging I realized that when trying to obtain the inode for the queue it was looking up at the open file descriptors from the lpwork thread, which of course would not share open file descriptors with main task.
Since this is working for Linux (I had copied this part of the porting layer from Linux) my reasoning is that SIGEV_THREAD is there implemented as a thread that is a child of the process which registered the signal handler and thus shares the open file descriptors. In NuttX this is implemented via a work queue so this is not the case.
Is my reading correct? If so, it may be worth adding this limitation to https://github.com/apache/incubator-nuttx/issues/1352
Best,
Matias
Re: limitation in SIGEV_THREAD?
Posted by "Matias N." <ma...@imap.cc>.
Thanks, I'll add a comment on that issue for reference.
On Tue, Jan 26, 2021, at 20:12, Gregory Nutt wrote:
> Yes, you are right. Perhaps this could be added to
> https://github.com/apache/incubator-nuttx/issues/1352
>
> That is a different issue description but might still be a good location
> because both issues have the same solution. The solution to both issues
> is to enhance the OS by adding a new OS facility to start a pthread from
> within OS just as if pthread_create() were called from that application
> task. That is not really that difficult. The SIGEV_THREAD logic would
> then run on that pthread.
>
> This same facility is also needed to correct the implementation of the
> AIO APIs. They also should run on pthreads as well. However, I am not
> aware of any functional shortcoming of the current AIO implementation
> that would justify the change.
>
> The only unattractive thing about this solution is that is does require
> more resources and is less efficient in general.
>
> Greg
>
> On 1/26/2021 4:26 PM, Matias N. wrote:
> > Hi,
> > working with nimBLE I found that I had an issue when scheduling a callback to be made from within the signal handler for a timer, which was set with SIGEV_THREAD. The issue was that I was pushing to a POSIX queue from within the handler, and it was failing with BADFD. From debugging I realized that when trying to obtain the inode for the queue it was looking up at the open file descriptors from the lpwork thread, which of course would not share open file descriptors with main task.
> >
> > Since this is working for Linux (I had copied this part of the porting layer from Linux) my reasoning is that SIGEV_THREAD is there implemented as a thread that is a child of the process which registered the signal handler and thus shares the open file descriptors. In NuttX this is implemented via a work queue so this is not the case.
> >
> > Is my reading correct? If so, it may be worth adding this limitation to https://github.com/apache/incubator-nuttx/issues/1352
> >
> > Best,
> > Matias
>
Re: limitation in SIGEV_THREAD?
Posted by "Matias N." <ma...@imap.cc>.
I'm thinking again about this. Why wouldn't it be possible to make functions
using SIGEV_THREAD (such as timer_settime) create a pthread behind
the scenes (only the first time a SIGEV_THREAD is setup)? The underlying
watchdog would go to a handler that posts a semaphore/condition variable
that the helper thread is waiting on. When this thread unblocks, it calls the
user handler directly.
The thread would be created per-process in KERNEL mode, so that shouldn't
be a problem (inside same address space as user handler). I suspect the
unblocking of the thread should also be possible in multi-process somehow
(named semaphore?).
This is essentially what I'm doing myself around a call to timer_settime.
Best,
Matias
On Wed, Jan 27, 2021, at 15:26, Gregory Nutt wrote:
>
> > Perhaps you could use a pool of application threads as is done with
> > the kernel threads for the low-priority work queue. So you could have
> > a small number of threads that service all tasks. When a user-space
> > thread is needed, it could be removed from the pool and be assigned to
> > the task to run the event. When the event processing completes, the
> > thread returned to the pool until it is again needed. Tasks could
> > wait for availability if there are no available threads in the pool.
> Nevermind! This would not work in KERNEL mode. In that case, each task
> (now better called processes) have there own separate protected address
> enviroments and threads could never be shared across processes. It
> would work fine in FLAT and PROTECTED modes where all tasks share the
> same address space.
>
Re: limitation in SIGEV_THREAD?
Posted by Gregory Nutt <sp...@gmail.com>.
> Perhaps you could use a pool of application threads as is done with
> the kernel threads for the low-priority work queue. So you could have
> a small number of threads that service all tasks. When a user-space
> thread is needed, it could be removed from the pool and be assigned to
> the task to run the event. When the event processing completes, the
> thread returned to the pool until it is again needed. Tasks could
> wait for availability if there are no available threads in the pool.
Nevermind! This would not work in KERNEL mode. In that case, each task
(now better called processes) have there own separate protected address
enviroments and threads could never be shared across processes. It
would work fine in FLAT and PROTECTED modes where all tasks share the
same address space.
Re: limitation in SIGEV_THREAD?
Posted by Gregory Nutt <sp...@gmail.com>.
> All global C++ objects are constructed in init task, so if these objects
> call open, fopen or fstream related API, the similar issue happens too. How
> can we fix this issue?
In order to do that, I think that each application task would have to
separately linked as they are with ELF modules or in KERNEL mode. There
is no issue in those cases.
> BTW, is it acceptable to create a pthread in each task to
> process SIGEV_THREAD dedicatedly in the small embedded device? Can we find
> a lightweight solution to fix it?
A straight-forward solution is easy, but a lightweight solution is not
so easy, depending upon what you consider to be lightweight. It is
difficult to imagine doing that with less then 1-2Kb or RAM for stack
and OS structures.
Perhaps you could use a pool of application threads as is done with the
kernel threads for the low-priority work queue. So you could have a
small number of threads that service all tasks. When a user-space
thread is needed, it could be removed from the pool and be assigned to
the task to run the event. When the event processing completes, the
thread returned to the pool until it is again needed. Tasks could wait
for availability if there are no available threads in the pool.
But that is not so easy.
RE: limitation in SIGEV_THREAD?
Posted by Xiang Xiao <xi...@gmail.com>.
ELF solution has to load the code to RAM, the overhead is too big to use for XIP system.
> -----Original Message-----
> From: Matias N. <ma...@imap.cc>
> Sent: Thursday, January 28, 2021 2:17 AM
> To: dev@nuttx.apache.org
> Subject: Re: limitation in SIGEV_THREAD?
>
> I think the independent ELF solution would again not be very good for constrained systems.
> The again, reading about SIGEV_THREAD makes me think that the specs are very loose in terms of what is guaranteed by the thread
running
> the callback.
>
> These kind of issues (and the fact that mixing signals with threads is not good) is what I think makes timerfd/signalfd/eventfd
attractive.
> Having those the need for these kind of notifications is greatly diminished, as you simply poll/select on your own thread, and it
can be done
> for other kind of fds simultaneously.
>
> Best,
> Matias
>
> On Wed, Jan 27, 2021, at 15:06, Gregory Nutt wrote:
> >
> > > My thinking is that maybe upon setting up the first SIGEV_THREAD notification for a task, a child thread would be created,
which would
> act as a worker, waiting on some semaphore for example. The bad thing is that it would linger until task is finished (not sure
even how easy
> would be to ensure it is deleted on task exit).
> >
> > This would be difficult because in PROTECTED and KERNEL modes, the OS
> > has no knowledge of any user-space symbols. Creating a pthread is
> > different from starting a task because you have to the user-space
> > address of the pthread entry point. That is not knowable in the cases
> > where the OS is separately linked. So this could not be done by the
> > core OS as a general solution. It could be done in some library
> > function if we had a user-space crt0.o to start the task.
> >
> > There is already a crt0.o for ELF modules and it could start such a thread.
> >
> > > Or is there a way for an lpwork thread to temporarily get into a task's environment?
> > No, because an LP thread is a kernel thread. It is privileged and if
> > it were to run user code in supervisor mode, that would be a major
> > security hole.
> > > BTW, for C++ global constructors I think there's also the issue that they are only called at boot and not each time a task
starts. At least I
> had that problem in the past.
> >
> > This would require a different build model with tasks (or at least
> > task-related constructors/destructors) that are separately built and a
> > crt0.o to start any threads.
> >
> > So one solution to both problems is to make all tasks into separately
> > linked ELF modules.
> >
> >
> >
Re: limitation in SIGEV_THREAD?
Posted by "Matias N." <ma...@imap.cc>.
I think the independent ELF solution would again not be very good for constrained systems.
The again, reading about SIGEV_THREAD makes me think that the specs are very loose in terms of
what is guaranteed by the thread running the callback.
These kind of issues (and the fact that mixing signals with threads is not good) is what I think makes
timerfd/signalfd/eventfd attractive. Having those the need for these kind of notifications is greatly diminished,
as you simply poll/select on your own thread, and it can be done for other kind of fds simultaneously.
Best,
Matias
On Wed, Jan 27, 2021, at 15:06, Gregory Nutt wrote:
>
> > My thinking is that maybe upon setting up the first SIGEV_THREAD notification for a task, a child thread would be created, which would act as a worker, waiting on some semaphore for example. The bad thing is that it would linger until task is finished (not sure even how easy would be to ensure it is deleted on task exit).
>
> This would be difficult because in PROTECTED and KERNEL modes, the OS
> has no knowledge of any user-space symbols. Creating a pthread is
> different from starting a task because you have to the user-space
> address of the pthread entry point. That is not knowable in the cases
> where the OS is separately linked. So this could not be done by the
> core OS as a general solution. It could be done in some library
> function if we had a user-space crt0.o to start the task.
>
> There is already a crt0.o for ELF modules and it could start such a thread.
>
> > Or is there a way for an lpwork thread to temporarily get into a task's environment?
> No, because an LP thread is a kernel thread. It is privileged and if it
> were to run user code in supervisor mode, that would be a major security
> hole.
> > BTW, for C++ global constructors I think there's also the issue that they are only called at boot and not each time a task starts. At least I had that problem in the past.
>
> This would require a different build model with tasks (or at least
> task-related constructors/destructors) that are separately built and a
> crt0.o to start any threads.
>
> So one solution to both problems is to make all tasks into separately
> linked ELF modules.
>
>
>
Re: limitation in SIGEV_THREAD?
Posted by Gregory Nutt <sp...@gmail.com>.
> My thinking is that maybe upon setting up the first SIGEV_THREAD notification for a task, a child thread would be created, which would act as a worker, waiting on some semaphore for example. The bad thing is that it would linger until task is finished (not sure even how easy would be to ensure it is deleted on task exit).
This would be difficult because in PROTECTED and KERNEL modes, the OS
has no knowledge of any user-space symbols. Creating a pthread is
different from starting a task because you have to the user-space
address of the pthread entry point. That is not knowable in the cases
where the OS is separately linked. So this could not be done by the
core OS as a general solution. It could be done in some library
function if we had a user-space crt0.o to start the task.
There is already a crt0.o for ELF modules and it could start such a thread.
> Or is there a way for an lpwork thread to temporarily get into a task's environment?
No, because an LP thread is a kernel thread. It is privileged and if it
were to run user code in supervisor mode, that would be a major security
hole.
> BTW, for C++ global constructors I think there's also the issue that they are only called at boot and not each time a task starts. At least I had that problem in the past.
This would require a different build model with tasks (or at least
task-related constructors/destructors) that are separately built and a
crt0.o to start any threads.
So one solution to both problems is to make all tasks into separately
linked ELF modules.
Re: limitation in SIGEV_THREAD?
Posted by "Matias N." <ma...@imap.cc>.
My thinking is that maybe upon setting up the first SIGEV_THREAD notification for a task, a child thread would be created, which would act as a worker, waiting on some semaphore for example. The bad thing is that it would linger until task is finished (not sure even how easy would be to ensure it is deleted on task exit).
Linux also supports (possibly due to this reason) SIGEV_THREAD_ID, which lets you specify a thread ID to send a signal to, although this is stated to be used internally.
Or is there a way for an lpwork thread to temporarily get into a task's environment?
BTW, for C++ global constructors I think there's also the issue that they are only called at boot and not each time a task starts. At least I had that problem in the past.
Best,
Matias
On Wed, Jan 27, 2021, at 14:45, Xiang Xiao wrote:
> All global C++ objects are constructed in init task, so if these objects
> call open, fopen or fstream related API, the similar issue happens too. How
> can we fix this issue?
> BTW, is it acceptable to create a pthread in each task to
> process SIGEV_THREAD dedicatedly in the small embedded device? Can we find
> a lightweight solution to fix it?
> We have encountered this issue many times, and workaround by calling
> file_open directly from the userspace, but it is obviously not the right
> way to go.
>
> Thanks
> Xiang
>
> On Tue, Jan 26, 2021 at 3:12 PM Gregory Nutt <sp...@gmail.com> wrote:
>
> > Yes, you are right. Perhaps this could be added to
> > https://github.com/apache/incubator-nuttx/issues/1352
> >
> > That is a different issue description but might still be a good location
> > because both issues have the same solution. The solution to both issues
> > is to enhance the OS by adding a new OS facility to start a pthread from
> > within OS just as if pthread_create() were called from that application
> > task. That is not really that difficult. The SIGEV_THREAD logic would
> > then run on that pthread.
> >
> > This same facility is also needed to correct the implementation of the
> > AIO APIs. They also should run on pthreads as well. However, I am not
> > aware of any functional shortcoming of the current AIO implementation
> > that would justify the change.
> >
> > The only unattractive thing about this solution is that is does require
> > more resources and is less efficient in general.
> >
> > Greg
> >
> > On 1/26/2021 4:26 PM, Matias N. wrote:
> > > Hi,
> > > working with nimBLE I found that I had an issue when scheduling a
> > callback to be made from within the signal handler for a timer, which was
> > set with SIGEV_THREAD. The issue was that I was pushing to a POSIX queue
> > from within the handler, and it was failing with BADFD. From debugging I
> > realized that when trying to obtain the inode for the queue it was looking
> > up at the open file descriptors from the lpwork thread, which of course
> > would not share open file descriptors with main task.
> > >
> > > Since this is working for Linux (I had copied this part of the porting
> > layer from Linux) my reasoning is that SIGEV_THREAD is there implemented as
> > a thread that is a child of the process which registered the signal handler
> > and thus shares the open file descriptors. In NuttX this is implemented via
> > a work queue so this is not the case.
> > >
> > > Is my reading correct? If so, it may be worth adding this limitation to
> > https://github.com/apache/incubator-nuttx/issues/1352
> > >
> > > Best,
> > > Matias
> >
>
Re: limitation in SIGEV_THREAD?
Posted by Xiang Xiao <xi...@gmail.com>.
All global C++ objects are constructed in init task, so if these objects
call open, fopen or fstream related API, the similar issue happens too. How
can we fix this issue?
BTW, is it acceptable to create a pthread in each task to
process SIGEV_THREAD dedicatedly in the small embedded device? Can we find
a lightweight solution to fix it?
We have encountered this issue many times, and workaround by calling
file_open directly from the userspace, but it is obviously not the right
way to go.
Thanks
Xiang
On Tue, Jan 26, 2021 at 3:12 PM Gregory Nutt <sp...@gmail.com> wrote:
> Yes, you are right. Perhaps this could be added to
> https://github.com/apache/incubator-nuttx/issues/1352
>
> That is a different issue description but might still be a good location
> because both issues have the same solution. The solution to both issues
> is to enhance the OS by adding a new OS facility to start a pthread from
> within OS just as if pthread_create() were called from that application
> task. That is not really that difficult. The SIGEV_THREAD logic would
> then run on that pthread.
>
> This same facility is also needed to correct the implementation of the
> AIO APIs. They also should run on pthreads as well. However, I am not
> aware of any functional shortcoming of the current AIO implementation
> that would justify the change.
>
> The only unattractive thing about this solution is that is does require
> more resources and is less efficient in general.
>
> Greg
>
> On 1/26/2021 4:26 PM, Matias N. wrote:
> > Hi,
> > working with nimBLE I found that I had an issue when scheduling a
> callback to be made from within the signal handler for a timer, which was
> set with SIGEV_THREAD. The issue was that I was pushing to a POSIX queue
> from within the handler, and it was failing with BADFD. From debugging I
> realized that when trying to obtain the inode for the queue it was looking
> up at the open file descriptors from the lpwork thread, which of course
> would not share open file descriptors with main task.
> >
> > Since this is working for Linux (I had copied this part of the porting
> layer from Linux) my reasoning is that SIGEV_THREAD is there implemented as
> a thread that is a child of the process which registered the signal handler
> and thus shares the open file descriptors. In NuttX this is implemented via
> a work queue so this is not the case.
> >
> > Is my reading correct? If so, it may be worth adding this limitation to
> https://github.com/apache/incubator-nuttx/issues/1352
> >
> > Best,
> > Matias
>
Re: limitation in SIGEV_THREAD?
Posted by Gregory Nutt <sp...@gmail.com>.
Yes, you are right. Perhaps this could be added to
https://github.com/apache/incubator-nuttx/issues/1352
That is a different issue description but might still be a good location
because both issues have the same solution. The solution to both issues
is to enhance the OS by adding a new OS facility to start a pthread from
within OS just as if pthread_create() were called from that application
task. That is not really that difficult. The SIGEV_THREAD logic would
then run on that pthread.
This same facility is also needed to correct the implementation of the
AIO APIs. They also should run on pthreads as well. However, I am not
aware of any functional shortcoming of the current AIO implementation
that would justify the change.
The only unattractive thing about this solution is that is does require
more resources and is less efficient in general.
Greg
On 1/26/2021 4:26 PM, Matias N. wrote:
> Hi,
> working with nimBLE I found that I had an issue when scheduling a callback to be made from within the signal handler for a timer, which was set with SIGEV_THREAD. The issue was that I was pushing to a POSIX queue from within the handler, and it was failing with BADFD. From debugging I realized that when trying to obtain the inode for the queue it was looking up at the open file descriptors from the lpwork thread, which of course would not share open file descriptors with main task.
>
> Since this is working for Linux (I had copied this part of the porting layer from Linux) my reasoning is that SIGEV_THREAD is there implemented as a thread that is a child of the process which registered the signal handler and thus shares the open file descriptors. In NuttX this is implemented via a work queue so this is not the case.
>
> Is my reading correct? If so, it may be worth adding this limitation to https://github.com/apache/incubator-nuttx/issues/1352
>
> Best,
> Matias