cross-process and cross-thread file locking

[thomas bonfort <th...@gmail.com>]

Hi all, sorry in advance if this is a dumb question.

The apr documentation for apr_file_lock states "Locks are established
on a per-thread/process basis; a second lock by the same thread will
not block." but this is not the behavior I am seeing. As apr_file_lock
on unix uses fcntl by default, a second lock by another thread of the
same process will not lock either.

I was using apr_file_lock as I need all my httpd threads/process to be
synchronized on an named ressource, and chose to create a lockfile
who's filename matches my named ressource. This does not work as with
a multi-threaded mpm the threads of the same process that created the
lockfile will not block on the call to apr_file_lock call.

>From my readings, it seems that file locking is a hazardous task to
get right, so what are my options to attain my goal:

- use my own implementation mimicking apr_file_lock, but that
unconditionnaly uses flock() instead of fcntl() ? I suspect that this
would not be a safe solution as some platforms fall back to fcntl for
flock.
- I tried using a posix semaphore which worked quite well, except in
the cases where either the process crashed or was terminated by httpd
because of a Timeout, and in that case the semaphore is never released
until a server reboot or manually messing in /dev/shm. If I attach a
cleanup call to the request pool, will it be called in the case where
the process is terminated after the Timeout delay ?

thank you for reading up to here,

Thomas

Re: cross-process and cross-thread file locking

[thomas bonfort <th...@gmail.com>]

On Thu, Sep 29, 2011 at 17:55, Ben Noordhuis <in...@bnoordhuis.nl> wrote:
> On Thu, Sep 29, 2011 at 16:54, thomas bonfort <th...@gmail.com> wrote:
>> Hi all, sorry in advance if this is a dumb question.
>>
>> The apr documentation for apr_file_lock states "Locks are established
>> on a per-thread/process basis; a second lock by the same thread will
>> not block." but this is not the behavior I am seeing. As apr_file_lock
>> on unix uses fcntl by default, a second lock by another thread of the
>> same process will not lock either.
>
> You're probably running into (POSIX mandated!) behaviour that requires
> that when a process closes a file descriptor for file X, *all* locks
> for X held by that process are released.

I also thought that might be the issue in the beginning, but after
double checking this also happens if the file handle is never closed.

>
> Absolutely brain dead. I can't begin to fathom the mind that thought it up.
>
>> I was using apr_file_lock as I need all my httpd threads/process to be
>> synchronized on an named ressource, and chose to create a lockfile
>> who's filename matches my named ressource. This does not work as with
>> a multi-threaded mpm the threads of the same process that created the
>> lockfile will not block on the call to apr_file_lock call.
>>
>> From my readings, it seems that file locking is a hazardous task to
>> get right, so what are my options to attain my goal:
>>
>> - use my own implementation mimicking apr_file_lock, but that
>> unconditionnaly uses flock() instead of fcntl() ? I suspect that this
>> would not be a safe solution as some platforms fall back to fcntl for
>> flock.
>
> flock() is not available on SunOS and it has nasty fork() semantics:
> process acquires lock, forks, child releases lock, parent loses lock
> (without getting told). Once again, brain dead.
>
> You also cannot rely on it working correctly (or at all) on NFS
> mounts. That's not really flock()'s fault, it's a shortcoming of the
> NFS protocol. fcntl() and lock() have the same issue.
>
> In my experience, the most reliable and portable approach is to create
> a lock file with open(O_CREAT|O_EXCL) that you unlink() afterwards. On
> EEXIST, sleep for a bit and try again.

that's the workaround I am currently using, but I would have hoped for
a cleaner solution (not using files at all if that is possible)

>
>> - I tried using a posix semaphore which worked quite well, except in
>> the cases where either the process crashed or was terminated by httpd
>> because of a Timeout, and in that case the semaphore is never released
>> until a server reboot or manually messing in /dev/shm. If I attach a
>> cleanup call to the request pool, will it be called in the case where
>> the process is terminated after the Timeout delay ?
>
> I don't think you can guarantee that your cleanup action always runs.
> If a worker process hangs, the master will eventually send it a
> SIGKILL.
>

Re: cross-process and cross-thread file locking

[thomas bonfort <th...@gmail.com>]

here's the tiny test program I wrote that shows the incorrect behavior
of apr_file_lock vs. flock :

if you compile with "gcc testlock.c -I/usr/include/apr-1.0 -lapr-1",
the output is:
thread aquired lock
thread aquired lock
thread released lock
thread released lock

adding -DUSE_FLOCK to the compilation uses flock instead of
apr_file_lock, the expected output is:
thread aquired lock
thread released lock
thread aquired lock
thread released lock

As the file is never closed, I would say that either the apr
documentation or implementation is flawed for apr_file_lock.

best regards,

thomas




On Thu, Sep 29, 2011 at 17:55, Ben Noordhuis <in...@bnoordhuis.nl> wrote:
> On Thu, Sep 29, 2011 at 16:54, thomas bonfort <th...@gmail.com> wrote:
>> Hi all, sorry in advance if this is a dumb question.
>>
>> The apr documentation for apr_file_lock states "Locks are established
>> on a per-thread/process basis; a second lock by the same thread will
>> not block." but this is not the behavior I am seeing. As apr_file_lock
>> on unix uses fcntl by default, a second lock by another thread of the
>> same process will not lock either.
>
> You're probably running into (POSIX mandated!) behaviour that requires
> that when a process closes a file descriptor for file X, *all* locks
> for X held by that process are released.
>
> Absolutely brain dead. I can't begin to fathom the mind that thought it up.
>
>> I was using apr_file_lock as I need all my httpd threads/process to be
>> synchronized on an named ressource, and chose to create a lockfile
>> who's filename matches my named ressource. This does not work as with
>> a multi-threaded mpm the threads of the same process that created the
>> lockfile will not block on the call to apr_file_lock call.
>>
>> From my readings, it seems that file locking is a hazardous task to
>> get right, so what are my options to attain my goal:
>>
>> - use my own implementation mimicking apr_file_lock, but that
>> unconditionnaly uses flock() instead of fcntl() ? I suspect that this
>> would not be a safe solution as some platforms fall back to fcntl for
>> flock.
>
> flock() is not available on SunOS and it has nasty fork() semantics:
> process acquires lock, forks, child releases lock, parent loses lock
> (without getting told). Once again, brain dead.
>
> You also cannot rely on it working correctly (or at all) on NFS
> mounts. That's not really flock()'s fault, it's a shortcoming of the
> NFS protocol. fcntl() and lock() have the same issue.
>
> In my experience, the most reliable and portable approach is to create
> a lock file with open(O_CREAT|O_EXCL) that you unlink() afterwards. On
> EEXIST, sleep for a bit and try again.
>
>> - I tried using a posix semaphore which worked quite well, except in
>> the cases where either the process crashed or was terminated by httpd
>> because of a Timeout, and in that case the semaphore is never released
>> until a server reboot or manually messing in /dev/shm. If I attach a
>> cleanup call to the request pool, will it be called in the case where
>> the process is terminated after the Timeout delay ?
>
> I don't think you can guarantee that your cleanup action always runs.
> If a worker process hangs, the master will eventually send it a
> SIGKILL.
>

Re: cross-process and cross-thread file locking

[Ben Noordhuis <in...@bnoordhuis.nl>]

On Thu, Sep 29, 2011 at 16:54, thomas bonfort <th...@gmail.com> wrote:
> Hi all, sorry in advance if this is a dumb question.
>
> The apr documentation for apr_file_lock states "Locks are established
> on a per-thread/process basis; a second lock by the same thread will
> not block." but this is not the behavior I am seeing. As apr_file_lock
> on unix uses fcntl by default, a second lock by another thread of the
> same process will not lock either.

You're probably running into (POSIX mandated!) behaviour that requires
that when a process closes a file descriptor for file X, *all* locks
for X held by that process are released.

Absolutely brain dead. I can't begin to fathom the mind that thought it up.

> I was using apr_file_lock as I need all my httpd threads/process to be
> synchronized on an named ressource, and chose to create a lockfile
> who's filename matches my named ressource. This does not work as with
> a multi-threaded mpm the threads of the same process that created the
> lockfile will not block on the call to apr_file_lock call.
>
> From my readings, it seems that file locking is a hazardous task to
> get right, so what are my options to attain my goal:
>
> - use my own implementation mimicking apr_file_lock, but that
> unconditionnaly uses flock() instead of fcntl() ? I suspect that this
> would not be a safe solution as some platforms fall back to fcntl for
> flock.

flock() is not available on SunOS and it has nasty fork() semantics:
process acquires lock, forks, child releases lock, parent loses lock
(without getting told). Once again, brain dead.

You also cannot rely on it working correctly (or at all) on NFS
mounts. That's not really flock()'s fault, it's a shortcoming of the
NFS protocol. fcntl() and lock() have the same issue.

In my experience, the most reliable and portable approach is to create
a lock file with open(O_CREAT|O_EXCL) that you unlink() afterwards. On
EEXIST, sleep for a bit and try again.

> - I tried using a posix semaphore which worked quite well, except in
> the cases where either the process crashed or was terminated by httpd
> because of a Timeout, and in that case the semaphore is never released
> until a server reboot or manually messing in /dev/shm. If I attach a
> cleanup call to the request pool, will it be called in the case where
> the process is terminated after the Timeout delay ?

I don't think you can guarantee that your cleanup action always runs.
If a worker process hangs, the master will eventually send it a
SIGKILL.