You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Pete Gonzalez <pg...@bluel.com> on 2005/02/14 11:06:56 UTC

Problems with /dev/random

We recently encountered a problem where svnserve was hanging
in the random number generator.  (Apparently someone else
encountered this problem and logged it as Debian Bug#285708.)

The culprit was a Linux daemon that reads from "/dev/urandom"
frequently.  Although "/dev/urandom" returns immediately, it
depletes the kernel entropy pool, causing "/dev/random" reads
to stall indefinitely.  I solved the problem by hacking the
Subversion apr_private.h DEV_RANDOM to read from "/dev/urandom"
instead of "/dev/random".

Since this problem took several hours to figure out, I'd like
to propose a few potential improvements:

1. Document this somewhere conspicuous

2. Make the default "/dev/urandom" (or a pseudorandom generator)
    instead of "/dev/random".

3. Add a "configure" switch enabling admins to specify whether
    or not they want to make everyone wait for the kernel to
    generate random numbers.  (In our scenario, this is ridiculous.)

Cheers,
-Pete


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Problems with /dev/random

Posted by Branko Čibej <br...@xbc.nu>.
Pete Gonzalez wrote:

> We recently encountered a problem where svnserve was hanging
> in the random number generator.  (Apparently someone else
> encountered this problem and logged it as Debian Bug#285708.)
>
> The culprit was a Linux daemon that reads from "/dev/urandom"
> frequently.  Although "/dev/urandom" returns immediately, it
> depletes the kernel entropy pool, causing "/dev/random" reads
> to stall indefinitely.  I solved the problem by hacking the
> Subversion apr_private.h DEV_RANDOM to read from "/dev/urandom"
> instead of "/dev/random".
>
> Since this problem took several hours to figure out, I'd like
> to propose a few potential improvements:
>
> 1. Document this somewhere conspicuous

It is quite conspiciously documented for APR, in the FAQ and in the Vook

> 2. Make the default "/dev/urandom" (or a pseudorandom generator)
>    instead of "/dev/random".

This is APR configuration. See apr's configure --with-devrandom.

> 3. Add a "configure" switch enabling admins to specify whether
>    or not they want to make everyone wait for the kernel to
>    generate random numbers.  (In our scenario, this is ridiculous.)

It's already there.

-- Brane


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: subversion _really_ slow

Posted by kf...@collab.net.
Greg Hudson <gh...@MIT.EDU> writes:
> Note that we wouldn't actually be using the new API; apr would be using
> it internally in apr_uuid_get().
> 
> >    * In the meantime
> 
> As we've done nothing about this problem for years, I don't see why it
> has suddenly become so urgent as to require difficult hacks to our
> source code while we wait for an apr fix.

Heh.  Well, if the hacks really are difficult, then it's not worth it.
If they are not difficult, then it is as worth it as it has ever been
(much the same could be said for many other bugs, after all).

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: subversion _really_ slow

Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2005-02-16 at 23:41, kfogel@collab.net wrote:
>    * Try to get the new api into APR, which we may be able to do
>      without too much trouble.

Note that we wouldn't actually be using the new API; apr would be using
it internally in apr_uuid_get().

>    * In the meantime

As we've done nothing about this problem for years, I don't see why it
has suddenly become so urgent as to require difficult hacks to our
source code while we wait for an apr fix.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: subversion _really_ slow

Posted by kf...@collab.net.
Greg Hudson <gh...@MIT.EDU> writes:
> On Wed, 2005-02-16 at 17:05, kfogel@collab.net wrote:
> > > This is an interesting point.  IMO the ideal solution would be to
> > > convince the APR people to fix their library, but if memory serves,
> > > the problem is from apr_generate_random_bytes() and only affects
> > > Subversion via the getuuid.c file.  If you look at the source code,
> > > writing a custom replacement for apr_uuid_get() is trivially easy and
> > > you could probably improve the implementation.
> 
> > Developers, would we want a patch as suggested by Pete Gonzalez above?
> > I'm inclined to say "yes", but would like others' thoughts.
> 
> The right answer is for APR to rev apr_generate_random_bytes() to take a
> flag saying whether PRNG data is okay, and for apr_uuid_get() to say
> that PRNG data is okay.  (If the system has no /dev/random, the flag
> shouldn't have any effect, in my opinion.)  I don't see any reason to
> address this problem within the Subversion code base, since I think it
> wouldn't be as easy.

I think you're right in the long run.  But the best strategy is
probably:

   * Try to get the new api into APR, which we may be able to do
     without too much trouble.

   * In the meantime, add some substitute functionality to Subversion,
     to use if that new APR symbol isn't defined.  That way we
     wouldn't require a bleeding edge APR to have this bug fixed right
     now, but we wouldn't have to forever keep code in Subversion that
     really belongs in APR.

I don't right now have time to *actually* do these things, but if we
are okay with this strategy, I could formally propose the APIs and
describe the implementations and at least (sigh) file an issue so we
don't forget.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: subversion _really_ slow

Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2005-02-16 at 17:05, kfogel@collab.net wrote:
> > This is an interesting point.  IMO the ideal solution would be to
> > convince the APR people to fix their library, but if memory serves,
> > the problem is from apr_generate_random_bytes() and only affects
> > Subversion via the getuuid.c file.  If you look at the source code,
> > writing a custom replacement for apr_uuid_get() is trivially easy and
> > you could probably improve the implementation.

> Developers, would we want a patch as suggested by Pete Gonzalez above?
> I'm inclined to say "yes", but would like others' thoughts.

The right answer is for APR to rev apr_generate_random_bytes() to take a
flag saying whether PRNG data is okay, and for apr_uuid_get() to say
that PRNG data is okay.  (If the system has no /dev/random, the flag
shouldn't have any effect, in my opinion.)  I don't see any reason to
address this problem within the Subversion code base, since I think it
wouldn't be as easy.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: subversion _really_ slow

Posted by kf...@collab.net.
[Developers, this is regarding the client's tendency to wait for a
long time on systems where APR is configured to use /dev/random
instead of /dev/urandom, but there isn't enough random data for what
Subversion requests at a given moment.]

Pete Gonzalez <pg...@bluel.com> writes:
> At 01:02 PM 2/16/2005, kfogel@collab.net wrote:
> >But, what if Subversion is sharing APR with other applications?  Can
> >Subversion just do whatever it wants with APR configuration?
> >Sometimes Subversion is linking against an existing APR, not building
> >its own.
> >(Of course, we could at least change our install instructions to
> >stipulate that if one *is* building APR just for Subversion, one
> >should configure with "--with-devrandom=/dev/urandom".)
> 
> This is an interesting point.  IMO the ideal solution would be to
> convince the APR people to fix their library, but if memory serves,
> the problem is from apr_generate_random_bytes() and only affects
> Subversion via the getuuid.c file.  If you look at the source code,
> writing a custom replacement for apr_uuid_get() is trivially easy and
> you could probably improve the implementation.

Or we could finally become multi-threaded, and spawn one thread to get
the random bytes from APR, while another one tries the custom
implementation if the first takes too long...

Mostly kidding there, but yes, you have a point :-).

Developers, would we want a patch as suggested by Pete Gonzalez above?
I'm inclined to say "yes", but would like others' thoughts.

The problem is real.  I've frustratingly experienced this lockup
myself, in fact.

-Karl

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: subversion _really_ slow

Posted by kf...@collab.net.
[Developers, this is regarding the client's tendency to wait for a
long time on systems where APR is configured to use /dev/random
instead of /dev/urandom, but there isn't enough random data for what
Subversion requests at a given moment.]

Pete Gonzalez <pg...@bluel.com> writes:
> At 01:02 PM 2/16/2005, kfogel@collab.net wrote:
> >But, what if Subversion is sharing APR with other applications?  Can
> >Subversion just do whatever it wants with APR configuration?
> >Sometimes Subversion is linking against an existing APR, not building
> >its own.
> >(Of course, we could at least change our install instructions to
> >stipulate that if one *is* building APR just for Subversion, one
> >should configure with "--with-devrandom=/dev/urandom".)
> 
> This is an interesting point.  IMO the ideal solution would be to
> convince the APR people to fix their library, but if memory serves,
> the problem is from apr_generate_random_bytes() and only affects
> Subversion via the getuuid.c file.  If you look at the source code,
> writing a custom replacement for apr_uuid_get() is trivially easy and
> you could probably improve the implementation.

Or we could finally become multi-threaded, and spawn one thread to get
the random bytes from APR, while another one tries the custom
implementation if the first takes too long...

Mostly kidding there, but yes, you have a point :-).

Developers, would we want a patch as suggested by Pete Gonzalez above?
I'm inclined to say "yes", but would like others' thoughts.

The problem is real.  I've frustratingly experienced this lockup
myself, in fact.

-Karl

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: subversion _really_ slow

Posted by Pete Gonzalez <pg...@bluel.com>.
At 01:02 PM 2/16/2005, kfogel@collab.net wrote:
>But, what if Subversion is sharing APR with other applications?  Can
>Subversion just do whatever it wants with APR configuration?
>Sometimes Subversion is linking against an existing APR, not building
>its own.
>(Of course, we could at least change our install instructions to
>stipulate that if one *is* building APR just for Subversion, one
>should configure with "--with-devrandom=/dev/urandom".)

This is an interesting point.  IMO the ideal solution would be to
convince the APR people to fix their library, but if memory serves,
the problem is from apr_generate_random_bytes() and only affects
Subversion via the getuuid.c file.  If you look at the source code,
writing a custom replacement for apr_uuid_get() is trivially easy and
you could probably improve the implementation.

Cheers,
-Pete


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: subversion _really_ slow

Posted by kf...@collab.net.
Pete Gonzalez <pg...@bluel.com> writes:
> If anything, this is an APR bug that should be reported to the
> Apache people.  But for Subversion's purposes, we can just fix
> the default APR configuration.

I agree that this is a baaaaad user experience.

But, what if Subversion is sharing APR with other applications?  Can
Subversion just do whatever it wants with APR configuration?
Sometimes Subversion is linking against an existing APR, not building
its own.

(Of course, we could at least change our install instructions to
stipulate that if one *is* building APR just for Subversion, one
should configure with "--with-devrandom=/dev/urandom".)


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: subversion _really_ slow

Posted by Pete Gonzalez <pg...@bluel.com>.
At 01:05 PM 2/15/2005, you wrote:
>I'm not going to claim I truly understand how /dev/random works, let alone 
>how
>it can be improperly configured.  However, I wonder if the misconfiguration
>can be detected programatically?

The /dev/random driver waits for unpredictable system events to
occur, i.e. timings between certain hardware interrupts.  (This
is why software like PGP will instruct the user to move the mouse
around or type some keys while the random number is being generated.)
If not enough events occur, then /dev/random will block and the
application has to wait.

On BSD systems, there is a command-line tool for configuring the
device, whereas in Linux it's controlled from /proc/sys/kernel/random/.
A "bad" configuration just means that it takes longer than necessary
to generate random numbers.  Of course, this only matters if there is
significant application for the "limited resource".

A second issue to consider is that on Linux 2.4 kernels, /dev/urandom
shares the same "entropy pool" as /dev/random.  This is a problem
because /dev/urandom is supposed to be an "unlimited" resource, but
on the 2.4 kernel it indirectly depletes /dev/random.  In our case,
/dev/random (and thus Subversion) was infinitely blocked as a result.

>In other words, is it possible to modify the Subversion or APR
>configure scripts to look for this situation and default to
>/dev/urandom if it appears necessary?

My opinion is that it is a mistake for APR to ever use /dev/random.
In an ideal configuration /dev/urandom is just as good, and even
if you could make some case that the degree of randomness matters
(which I would dispute), I cannot see how "Subversion really slow"
is an acceptable user experience.

If anything, this is an APR bug that should be reported to the
Apache people.  But for Subversion's purposes, we can just fix
the default APR configuration.

Cheers,
-Pete


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: subversion _really_ slow

Posted by Sean Laurent <se...@neuronfarm.com>.
On Tuesday 15 February 2005 11:39 am, Pete Gonzalez wrote:
> > > I currently have a problem with generating random numbers (/dev/random
> > takes
> > > undefinied amount of time) which slows things very much.
> > > That could explain the 'unpredictable' factor: maybe you don't have
> > > enough random numbers on your system. But do a strace and it'll help
> > > understand where the process 'hangs'.
> >
> >I recompiled APR with "--with-devrandom=/dev/urandom", then recompiled
> >my Subversion client against that APR, and the problem went away.
>
> I have thought about this problem, and I would like to argue that
> "/dev/urandom" (or even better the alternate timer-based implementation)
> should be the *default*, rather than being a retarded "gotcha" for
> new users:
<TEXT DELETED>
> In my case, the /dev/random issue turned out to be a design problem
> in the 2.4 Linux kernel (see below).  I've also seen several newsgroup
> discussions discussing improper default configurations for /dev/random,
> so this situation can occur in several ways.

I'm not going to claim I truly understand how /dev/random works, let alone how 
it can be improperly configured.  However, I wonder if the misconfiguration 
can be detected programatically?  In other words, is it possible to modify 
the Subversion or APR configure scripts to look for this situation and 
default to /dev/urandom if it appears necessary?

-Sean
------------------------------
Democracy is three wolves and a lamb voting on what to have for dinner. 
Liberty is a well-armed lamb contesting the vote.
-- Benjamin Franklin
------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: subversion _really_ slow

Posted by Pete Gonzalez <pg...@bluel.com>.
> > I currently have a problem with generating random numbers (/dev/random 
> takes
> > undefinied amount of time) which slows things very much.
> > That could explain the 'unpredictable' factor: maybe you don't have enough
> > random numbers on your system. But do a strace and it'll help understand
> > where the process 'hangs'.
>
>Oh, yeah, this happened to me too!
>
>I recompiled APR with "--with-devrandom=/dev/urandom", then recompiled
>my Subversion client against that APR, and the problem went away.

I have thought about this problem, and I would like to argue that
"/dev/urandom" (or even better the alternate timer-based implementation)
should be the *default*, rather than being a retarded "gotcha" for
new users:

This code is called from APR's getuuid.c (which generates the UUID's),
and the UUID spec referenced there does not require the numbers to be
truly random.

In my case, the /dev/random issue turned out to be a design problem
in the 2.4 Linux kernel (see below).  I've also seen several newsgroup
discussions discussing improper default configurations for /dev/random,
so this situation can occur in several ways.

Although some people will figure it out and begrudgingly recompile
their server, I think most admins will just conclude that Subversion
is inefficient by design.  Which, I argue, is not far from the truth
if /dev/random is kept as the default.  ;-)

Cheers,
-Pete


------------Forwarded Message-----------
Date: Tue, 15 Feb 2005 10:59:38 -0500
From: Theodore Ts'o <ty...@mit.edu>
To: Pete Gonzalez <pg...@bluel.com>
Subject: Re: Question regarding /dev/urandom design

 > I'm writing with a question regarding your random.c driver for the
 > Linux kernel.  It would appear that the intent of "/dev/urandom"
 > is to supply random numbers for applications that need immediate
 > results, and where quality can be sacrificed for quantity.  However,
 > it appears that when an application reads from /dev/urandom
 > frequently, other applications reading from /dev/random will hang
 > indefinitely because the entropy is being constantly depleted.
 >
 > If so, then IMO this defeats the purpose of "/dev/urandom", since
 > although it offers an unlimited resource for the calling application,
 > it is still a limited resource from the OS's perspective.  A possible
 > solution would be to maintain a separate entropy pool for
 > "/dev/urandom", or maybe somehow the "/dev/random" requests can
 > be prioritized over "/dev/urandom".

It is a separate entropy pool in 2.6, but people still use it wrong,
because they seem to use it instead of a cryptographic random
generator, which is what most of them *really* seem to want.  All you
should do is read 16 bytes from /dev/random, and use it to seed a SHA
based random number generator --- which you use in userspace!  It's
faster, and really the right answer.

                                                 - Ted


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org