You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Evgeny Kotkov <ev...@visualsvn.com> on 2015/12/30 16:49:08 UTC

[RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Evgeny Kotkov <ev...@visualsvn.com> writes:

> Stefan Fuhrmann <st...@apache.org> writes:
>
>> Thanks, it would be nice if could do something within Subversion because
>> httpd and apr patches may take a while to trickle down into releases.
>> Meanwhile, I posted a patch for mod_dav on the httpd dev list.
>
> I took a look at what's happening, and I think the root cause of the problem
> is that during a PROPFIND walk, we use the request pool for allocations that
> happen per every walked entry.  This occurs when mod_dav_svn/repos.c:
> do_walk() calls dav_propfind_walker() with dav_walk_resource.pool pointing
> to the request pool.

The problem with unbounded memory usage during PROPFINDs in mod_dav is old
and quite fundamental.  Its first acknowledgment dates back to year 2003 [1],
and the corresponding Bugzilla issue has been open since 2009 [2, 3].

Subversion's mod_dav_svn module is built around mod_dav, and is affected as
well.  The propfind walker makes different FS calls that allocate a lot of
memory — especially if a cache miss happens.  A typical walk can cross the
mod_dav - mod_dav_svn boundary multiple times.  Since mod_dav was written
long ago and doesn't follow the same pool usage practices as Subversion,
some of the allocations happen in long-living pools, such as the request
pool.

So, the memory usage for a PROPFIND of depth 1 (svn ls -v) is unbounded.
Although the actual memory consumption can be mitigated by the cache hits,
since a cache hit either reduces or eliminates additional allocations, a real
server can easily consume gigabytes of RAM, when the data is not and possibly
can't fit in the cache.  The problem is serious, and can lead to various out
of memory conditions, such as constant swapping or server crashes.

I'm aware of two possible ways of solving the problem:

 (1) Fix mod_dav, adjust mod_dav_svn accordingly

 (2) Reimplement PROPFIND in mod_dav_svn

Doing (1) is hard, since it requires revving mod_dav's API and special care
for other modules using it.  As far as I know, those modules don't have tests
and not breaking their behavior could be a challenging task.

I believe that currently we don't have the necessary resources (aka committers)
to do that.  Given that the problem exists since 2003, I also doubt that it's
going to solve itself in the nearby future.  And moreover, even if somehow
managed to do that, the fix would probably only be available in the newer
2.6.x httpd releases.

With that in mind, I propose we do (2).

Although I'm not entirely happy with reimplementing a part of the protocol, I
don't think that there are other realistic approaches to this serious problem.
It wouldn't be the first time we're doing something like this, since we already
handle POST ourselves, as well as provide the custom Subversion's HTTPv2
protocol implementation.

Apart from solving the memory usage issue, having the PROPFIND implementation
in mod_dav_svn would allow us to improve a couple of other aspects of how we
handle these requests, e.g.:

 - Shrink down the PROPFIND response size by avoiding duplicate xmlns:
   namespace declarations that currently exist.

 - Optimize how we retrieve properties from the FS layer, since we would
   no longer be doing mod_dav - mod_dav_svn - mod_dav transitions, and
   would have more flexibility.

I attached a proof-of-concept patch that does (2).  This patch solves the
memory usage problem in my experiments and passes HTTPv2 / HTTPv1 tests.

Thoughts?

[1] https://mail-archives.apache.org/mod_mbox/httpd-dev/200305.mbox/%3C86smqzgcuf.fsf@kepler.ch.collab.net%3E
[2] https://bz.apache.org/bugzilla/show_bug.cgi?id=52123
[3] https://bz.apache.org/bugzilla/show_bug.cgi?id=48130
[4] https://mail-archives.apache.org/mod_mbox/subversion-users/201512.mbox/%3C1CEE115D02633942A40D49D447DCF46E432F3A32@SD01CFMM0202.OMEGA.DCE-EIR.NET%3E


Regards,
Evgeny Kotkov

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 15 January 2016 at 09:26, Daniel Shahaf <da...@apache.org> wrote:
> Evgeny Kotkov wrote on Thu, Jan 14, 2016 at 18:40:11 +0300:
>> Not quite sure on how do we continue from here.
>
> Well, it isn't consensus, but there's an old fallback we can use:
> pick an agreed-upon third party and let him decide.
>
> So let's ask a mod_dav/mod_dav_svn dev for his opinion and do what he says.
>
I am not a mod_dav developer, but I've been hacking on mod_dav_svn for
quite a while now.

In my opinion we should start reimplemeting mod_dav in mod_dav_svn
like we already did with implementing our own HTTPv2 protocol instead
of DAV and move forward. As far as I remember we discussed this
approach with Ben Reser two years ago on SVN Hackathon in Berlin and
he said that this might be good idea.

-- 
Ivan Zhakov

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Evgeny Kotkov wrote on Fri, Jan 29, 2016 at 17:37:28 +0300:
> Daniel Shahaf <da...@apache.org> writes:
> 
> > Well, it isn't consensus, but there's an old fallback we can use:
> > pick an agreed-upon third party and let him decide.
> >
> > So let's ask a mod_dav/mod_dav_svn dev for his opinion and do what he says.
> 
> It seems like we still don't have a consensus here.  I still say we should
> just go with (2) and have the problem solved.  But since we were unable to
> agree on that, this issue is now tracked as SVN-4616:
> 
>   https://issues.apache.org/jira/browse/SVN-4616

I was merely trying to help find a find a better solution by reviewing
a technical proposal that had been brought up to the list.  It was never
my intention to cause the issue to be left open indefinitely.

What can we do to make progress on fixing this bug?

> One thing worth adding is that there's also a problem with how we log actions
> from mod_dav_svn callbacks.
> 
> Within a callback, we lack the information about the state of the operation.
> As one example, deadprops.c:db_first_name() calls dav_svn__operational_log().
> While handling <allprop/> depth 1 PROPFIND requests, db_first_name() is called
> once per every walked entry.  Every subsequent call rewrites previously set
> r->subprocess_env values.  Hence, the actual log entry is determined by the
> last walked entry, and that's incorrect.  And all these values are allocated
> in the request pool, thus triggering unbounded memory usage.
> 
> I think that this part of the problem can't be solved properly without
> reimplementing the PROPFIND in mod_dav_svn.

What does "properly solved" mean to you?  I expect a 'propget
--depth=files $URL' to result in one operational log entry for $URL (and
no operational log entries for the children).

For the short term, we can simply skip setting the "SVN-ACTION" value
and friends, if a value is already set.  That will stop the O(#dirents)
memory usage.  We can do this in the next patch release.

For the slightly longer term, perhaps we should look into moving the
dav_svn__opertional_log() call from db_first_name() to mod_dav_svn/repos.c:walk():
that function seems like a more logical place to log a "I am doing
a propget --depth=files" operation from.  (Can this be done without
extending the mod_dav↔walker API?)

Cheers,

Daniel

P.S. I may have limited net access next week, but I'll follow this
thread as much as I can.

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn

Posted by Stefan Fuhrmann <st...@apache.org>.
On 29.01.2016 15:37, Evgeny Kotkov wrote:
> Daniel Shahaf<da...@apache.org>  writes:
>
>> Well, it isn't consensus, but there's an old fallback we can use:
>> pick an agreed-upon third party and let him decide.
>>
>> So let's ask a mod_dav/mod_dav_svn dev for his opinion and do what he says.
> It seems like we still don't have a consensus here.  I still say we should
> just go with (2) and have the problem solved.  But since we were unable to
> agree on that, this issue is now tracked as SVN-4616:
>
>    https://issues.apache.org/jira/browse/SVN-4616
>
> One thing worth adding is that there's also a problem with how we log actions
> from mod_dav_svn callbacks.

Well, approving r1721285 and r1725180 will go a long
way reducing the impact of that issue for most users.
In particular, those setting MaxMemFree to "eat all my
memory".

> Within a callback, we lack the information about the state of the operation.
> As one example, deadprops.c:db_first_name() calls dav_svn__operational_log().
> While handling <allprop/> depth 1 PROPFIND requests, db_first_name() is called
> once per every walked entry.  Every subsequent call rewrites previously set
> r->subprocess_env values.  Hence, the actual log entry is determined by the
> last walked entry, and that's incorrect.  And all these values are allocated
> in the request pool, thus triggering unbounded memory usage.
>
> I think that this part of the problem can't be solved properly without
> reimplementing the PROPFIND in mod_dav_svn.
Having the problem fixed in mod_dav itself (e.g. see
https://bz.apache.org/bugzilla/show_bug.cgi?id=48130)
would make future maintenance "their problem". Forking
the code makes it "ours", more specifically: *yours*.

Given that the issue seems to be known for 5+ years
and patches for it are available for 3+ years now,
responsiveness is obviously bad. Did you contact them
before suggesting the fork and what was their response?

Concerning the fork, I think Ivan made a good point saying
that we already did something similar for httpv2. So, how
much of mod_dav_svn would we want to folk (property
handling only or all the remainder as well) and how much
code would that add to mod_dav_svn (50%?, more?, less?).

If the total addition is significantly smaller than mod_dav_svn,
maintaining it should be feasible after the first couple of
years of stabilization.

-- Stefan^2.

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Posted by Evgeny Kotkov <ev...@visualsvn.com>.
Daniel Shahaf <da...@apache.org> writes:

> Well, it isn't consensus, but there's an old fallback we can use:
> pick an agreed-upon third party and let him decide.
>
> So let's ask a mod_dav/mod_dav_svn dev for his opinion and do what he says.

It seems like we still don't have a consensus here.  I still say we should
just go with (2) and have the problem solved.  But since we were unable to
agree on that, this issue is now tracked as SVN-4616:

  https://issues.apache.org/jira/browse/SVN-4616

One thing worth adding is that there's also a problem with how we log actions
from mod_dav_svn callbacks.

Within a callback, we lack the information about the state of the operation.
As one example, deadprops.c:db_first_name() calls dav_svn__operational_log().
While handling <allprop/> depth 1 PROPFIND requests, db_first_name() is called
once per every walked entry.  Every subsequent call rewrites previously set
r->subprocess_env values.  Hence, the actual log entry is determined by the
last walked entry, and that's incorrect.  And all these values are allocated
in the request pool, thus triggering unbounded memory usage.

I think that this part of the problem can't be solved properly without
reimplementing the PROPFIND in mod_dav_svn.


Regards,
Evgeny Kotkov

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Posted by Daniel Shahaf <da...@apache.org>.
Evgeny Kotkov wrote on Thu, Jan 14, 2016 at 18:40:11 +0300:
> Not quite sure on how do we continue from here.

Well, it isn't consensus, but there's an old fallback we can use:
pick an agreed-upon third party and let him decide.

So let's ask a mod_dav/mod_dav_svn dev for his opinion and do what he says.

Cheers,

Daniel

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Posted by Evgeny Kotkov <ev...@visualsvn.com>.
Daniel Shahaf <d....@daniel.shahaf.name> writes:

>> I believe that (1) is much harder than (2).
>>
>> The rewrite of mod_dav pool's usage is a recipe for rarely reproducible
>> crashes caused by lifetime issues.  The functions and callbacks often modify
>> or attach allocated data to other objects.  We'd also need to deal with
>> per-object subpools like propdb->p, ensure that errors survive up to the
>> proper point of time, provide different API versions compatible in terms of
>> the objects' lifetimes, hope that nobody — API users and mod_dav itself —
>> accidentally relies on something to be allocated in a long-living pool, etc.
>>
>> Frankly, I don't think that this is a realistic approach to the problem.
>>
>
> All the problems you describe are generic problems that apply to *any*
> rewrite of *any* C codebase to change its dynamic memory allocation
> patterns.  Your argument could be applied to *any* case where somebody
> proposes to change some library's object lifetime patterns.  It implies
> that it is impossible or infeasible to patch a library to have tighter
> lifetime promises.

This doesn't make them less applicable to the scope of what needs to be
done for (1), especially with the specifics of mod_dav's code in mind.

And no, I wasn't implying that it's impossible.  What I'm trying to say is
that I don't see it as a viable solution for the particular problem due to it

 - being far more complex than (2)

 - having a higher chance of introducing severe issues that are hard to
   diagnose

 - requiring changes that could affect third-party code (other users of
   mod_dav's public API)

 - having a prerequisite of being very well acquainted with mod_dav's code

 - not solving the problem unless certain preconditions are met (that is,
   either using httpd/mod_dav 2.6.x, once it's released, or being dependent
   on a potentially destabilizing backport to 2.4.x / 2.2.x, or not using
   the officially released version, but a version with the custom patch
   applied)

As opposed to that, (2) is a relatively low-cost solution that also has
certain benefits, allowing us to send less data over the wire or to stop
fetching the same information multiple times from lower levels.

>> And we do have a history of breaking mod_dav (and then reverting changes)
>> with much smaller fixes.
>
> Would you be able to cite concrete examples?

I was referring to

  https://svn.apache.org/r1529559
  https://svn.apache.org/r1531505
  https://svn.apache.org/r1602338

> I think forking mod_dav into mod_dav_svn would be a mistake.
>
> I think you should patch mod_dav.
>
> Can I help with the latter approach?  E.g., by reaching out to the
> dev@httpd folks, or by coding / testing / reviewing a patch?  I will
> have limited time for this since it's not part of my dayjob.
>
> How do we move forward?

Thanks for offering your help.

I started this thread in order to draw attention to the problem and to check
whether what I propose squares with what everyone else is thinking.  I am
certainly not going to push (2) forward if people are unhappy with that.

On the other hand, I am confident that I won't be able to pull (1) off, so
I don't plan on doing that.  This leaves us with the serious problem that
probably is affecting a lot of users in their ordinary workflow.

Not quite sure on how do we continue from here.


Regards,
Evgeny Kotkov

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Evgeny Kotkov wrote on Wed, Jan 13, 2016 at 17:33:36 +0300:
> Daniel Shahaf <d....@daniel.shahaf.name> writes:
> 
> > I still do not see the problem.  mod_dav is not a huge amount of code,
> > and the work you describe does not sound too hard.  Why didn't your
> > attempt to patch mod_dav succeed?
> 
> [...]
> 
> > Option (1) does not have "unknown complexity".  According to what you
> > wrote above, it involves converting mod_dav to dual pools.  That is
> > a finite task: for each of the 90 functions and 73 callbacks, at each
> > callsite, determine what lifetime the object it returns or constructs
> > needs, find a pool of that lifetime, and pass that pool to the function
> > or callback.
> 
> I believe that (1) is much harder than (2).
> 
> The rewrite of mod_dav pool's usage is a recipe for rarely reproducible
> crashes caused by lifetime issues.  The functions and callbacks often modify
> or attach allocated data to other objects.  We'd also need to deal with
> per-object subpools like propdb->p, ensure that errors survive up to the
> proper point of time, provide different API versions compatible in terms of
> the objects' lifetimes, hope that nobody — API users and mod_dav itself —
> accidentally relies on something to be allocated in a long-living pool, etc.
> 
> Frankly, I don't think that this is a realistic approach to the problem.
> 

All the problems you describe are generic problems that apply to *any*
rewrite of *any* C codebase to change its dynamic memory allocation
patterns.  Your argument could be applied to *any* case where somebody
proposes to change some library's object lifetime patterns.  It implies
that it is impossible or infeasible to patch a library to have tighter
lifetime promises.

> And we do have a history of breaking mod_dav (and then reverting changes)
> with much smaller fixes.

Would you be able to cite concrete examples?

> > Why wouldn't they become part of 2.2.x?
> >
> > Stefan's patch is a one-line bugfix patch that applies cleanly, so
> > anyone supporting httpd-2.2 can easily apply it.
> 
> This one-line patch doesn't resolve the root cause of the problem.
> 
> As long as we allocate in the request pool on every iteration, the memory
> usage would always be unbounded, irrespectively of how we tweak the absolute
> values.  And even with these tweaks, I still see heavy memory consumption in
> my tests (details are included below).
> 
> I also think that if we had the necessary aspects of PROPFIND in our control,
> we probably wouldn't even get to the point where we find ourselves removing
> subpools, saying that they are the root cause of the problem or inspecting
> the intrinsics of the APR allocator.
> 
> The last thing worth mentioning is that there was no reaction even to this
> patch; mod_dav isn't being actively maintained these days.  I only counted
> about 35 dav-specific commits to modules/dav/main since 2005, and several
> of them were reverted afterwards.
> 

The "number of commits" statistic of a project is not a sensitive test
of its actively-maintainedness: a project may have few commits but be
actively maintained.  A better measure is whether "DEFECT"-kind bugs are
being addressed timely.  To my recollection, every bug we found in
mod_dav has been addressed.  (Admittedly, some of them were addressed by
an svn developer who earned httpd PMC membership in the process
(breser).)

> 
> 
> Regards,
> Evgeny Kotkov

So, to summarize:

My understanding is that you disqualified / ruled out the "patch mod_dav"
approach because mod_dav is a library.  (I might have misunderstood, of
course, in which case I'd appreciate being corrected.)

I think forking mod_dav into mod_dav_svn would be a mistake.

I think you should patch mod_dav.

Can I help with the latter approach?  E.g., by reaching out to the
dev@httpd folks, or by coding / testing / reviewing a patch?  I will
have limited time for this since it's not part of my dayjob.

How do we move forward?

Daniel

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn

Posted by Stefan Fuhrmann <st...@apache.org>.
On 13.01.2016 15:33, Evgeny Kotkov wrote:
> Stefan Fuhrmann <st...@apache.org> writes:
> If I replace apr_palloc() with malloc() / free() and patch mod_dav to avoid
> creating the propdb->p subpools, I still see inadequate memory consumption.
>
> The environment is close to the one reported by the user, and preparing a
> 4.5 MB response causes the heap commit size to slowly increase up to 70 MB.
> This is a lot, considering a possible amount of parallel requests and *can*
> cause problems in any reasonable setup.  It could as well be much worse with
> other possible environments and data.

Sure and I don't deny neither the existence nor the relevance
of the problem.  All I want to point out that there are two
levels of severity here - one easily consuming 10s of GB per
request and one using 100 MB or so for a directory with no user
properties at its entries.

The first is being addressed by my patches, the latter is basically
what you got with 1.6-style caching anyway and what you are
trying to solve.

>> Does the patch already include the improvements you suggested above?
>> Is it feasible to #ifdef the usage of that code?  I think if we have
>> an easy and clean opt-in / opt-out, people could be more accepting of
>> that workaround.
>
> The patch doesn't include these improvements.  I can drive the patch up to
> a committable state, commit it, and we can enhance these aspects later in
> trunk.  I attached an updated version of the patch without the regression
> tests that were committed in r1724103.

Having the option to reduce the server load and network
traffic would certainly be a plus.  plugins.svn.wordpress.org
is now closing it at 60k entries in the repo root.

> I don't really like the idea of surrounding this code with #ifdef-s, since
> I can't see a reason to conditionally turn it off, and not enabling it by
> default would leave us with undertested code or code that's never executed.

The point of the #ifdef would be an opt-out for the time the
problem got fixed on the mod_dav side.  I agree that making it
opt-in would be problematic on multiple levels.  Also, if it's
only a handful of #ifdef sections, then the change is reasonably
isolated from the rest, i.e. has a reduced perceived risk.

I still think we need to fix eventually fix mod_dav.  In the
long run, that should always be cheaper than maintaining a fork.

-- Stefan^2.

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Posted by Evgeny Kotkov <ev...@visualsvn.com>.
Daniel Shahaf <d....@daniel.shahaf.name> writes:

> I still do not see the problem.  mod_dav is not a huge amount of code,
> and the work you describe does not sound too hard.  Why didn't your
> attempt to patch mod_dav succeed?

[...]

> Option (1) does not have "unknown complexity".  According to what you
> wrote above, it involves converting mod_dav to dual pools.  That is
> a finite task: for each of the 90 functions and 73 callbacks, at each
> callsite, determine what lifetime the object it returns or constructs
> needs, find a pool of that lifetime, and pass that pool to the function
> or callback.

I believe that (1) is much harder than (2).

The rewrite of mod_dav pool's usage is a recipe for rarely reproducible
crashes caused by lifetime issues.  The functions and callbacks often modify
or attach allocated data to other objects.  We'd also need to deal with
per-object subpools like propdb->p, ensure that errors survive up to the
proper point of time, provide different API versions compatible in terms of
the objects' lifetimes, hope that nobody — API users and mod_dav itself —
accidentally relies on something to be allocated in a long-living pool, etc.
And we do have a history of breaking mod_dav (and then reverting changes)
with much smaller fixes.

Frankly, I don't think that this is a realistic approach to the problem.

> Why wouldn't they become part of 2.2.x?
>
> Stefan's patch is a one-line bugfix patch that applies cleanly, so
> anyone supporting httpd-2.2 can easily apply it.

This one-line patch doesn't resolve the root cause of the problem.

As long as we allocate in the request pool on every iteration, the memory
usage would always be unbounded, irrespectively of how we tweak the absolute
values.  And even with these tweaks, I still see heavy memory consumption in
my tests (details are included below).

I also think that if we had the necessary aspects of PROPFIND in our control,
we probably wouldn't even get to the point where we find ourselves removing
subpools, saying that they are the root cause of the problem or inspecting
the intrinsics of the APR allocator.

The last thing worth mentioning is that there was no reaction even to this
patch; mod_dav isn't being actively maintained these days.  I only counted
about 35 dav-specific commits to modules/dav/main since 2005, and several
of them were reverted afterwards.


Stefan Fuhrmann <st...@apache.org> writes:

> The "designed" memory usage of mod_dav as in "what gets allocated via palloc"
> is O(total size of all properties).  That is unbounded and if you attach
> large properties to each node in a large folder, you may end up with GB
> sized pools.  This is a real problem and needs to fixed eventually.
>
> The current situation, however, is that we have O(N^2) allocation *overhead*,
> i.e. memory provided by the APR pool allocator that will never be used.
>
> So, it is not that anybody allocates large structures in long-living pools
> (except for large properties).  Instead APR reuses blocks from large,
> short-lived allocations for small long-lived allocations.  Preventing
> that already goes a long way mitigating the problems with any reasonable
> setup.

If I replace apr_palloc() with malloc() / free() and patch mod_dav to avoid
creating the propdb->p subpools, I still see inadequate memory consumption.

The environment is close to the one reported by the user, and preparing a
4.5 MB response causes the heap commit size to slowly increase up to 70 MB.
This is a lot, considering a possible amount of parallel requests and *can*
cause problems in any reasonable setup.  It could as well be much worse with
other possible environments and data.

> Assuming that your analysis for (1) is correct, I think (2) is a viable
> option.  However, I would very much like to see mod_dav fixed as well and
> to have (2) only as a temporary workaround.  "Temporary" being a quite
> euphemistic term here ...

[...]

> Does the patch already include the improvements you suggested above?
> Is it feasible to #ifdef the usage of that code?  I think if we have
> an easy and clean opt-in / opt-out, people could be more accepting of
> that workaround.

The patch doesn't include these improvements.  I can drive the patch up to
a committable state, commit it, and we can enhance these aspects later in
trunk.  I attached an updated version of the patch without the regression
tests that were committed in r1724103.

I don't really like the idea of surrounding this code with #ifdef-s, since
I can't see a reason to conditionally turn it off, and not enabling it by
default would leave us with undertested code or code that's never executed.


Regards,
Evgeny Kotkov

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Posted by Stefan Fuhrmann <st...@apache.org>.
On 30.12.2015 16:49, Evgeny Kotkov wrote:
> Evgeny Kotkov <ev...@visualsvn.com> writes:
>
>> Stefan Fuhrmann <st...@apache.org> writes:
>>
>>> Thanks, it would be nice if could do something within Subversion because
>>> httpd and apr patches may take a while to trickle down into releases.
>>> Meanwhile, I posted a patch for mod_dav on the httpd dev list.
>>
>> I took a look at what's happening, and I think the root cause of the problem
>> is that during a PROPFIND walk, we use the request pool for allocations that
>> happen per every walked entry.  This occurs when mod_dav_svn/repos.c:
>> do_walk() calls dav_propfind_walker() with dav_walk_resource.pool pointing
>> to the request pool.
>
> The problem with unbounded memory usage during PROPFINDs in mod_dav is old
> and quite fundamental.  Its first acknowledgment dates back to year 2003 [1],
> and the corresponding Bugzilla issue has been open since 2009 [2, 3].
>
> Subversion's mod_dav_svn module is built around mod_dav, and is affected as
> well.  The propfind walker makes different FS calls that allocate a lot of
> memory — especially if a cache miss happens.  A typical walk can cross the
> mod_dav - mod_dav_svn boundary multiple times.  Since mod_dav was written
> long ago and doesn't follow the same pool usage practices as Subversion,
> some of the allocations happen in long-living pools, such as the request
> pool.

Given your response to Daniel later in this thread,
I want to re-iterate the nature of the problem to
prevent decision making based on false mental models.

The "designed" memory usage of mod_dav as in "what
gets allocated via palloc" is O(total size of all
properties).  That is unbounded and if you attach
large properties to each node in a large folder,
you may end up with GB sized pools.  This is a real
problem and needs to fixed eventually.

The current situation, however, is that we have O(N^2)
allocation *overhead*, i.e. memory provided by the
APR pool allocator that will never be used.

So, it is not that anybody allocates large structures
in long-living pools (except for large properties).
Instead APR reuses blocks from large, short-lived
allocations for small long-lived allocations.
Preventing that already goes a long way mitigating
the problems with any reasonable setup.

> So, the memory usage for a PROPFIND of depth 1 (svn ls -v) is unbounded.
> Although the actual memory consumption can be mitigated by the cache hits,
> since a cache hit either reduces or eliminates additional allocations, a real
> server can easily consume gigabytes of RAM, when the data is not and possibly
> can't fit in the cache.  The problem is serious, and can lead to various out
> of memory conditions, such as constant swapping or server crashes.
>
> I'm aware of two possible ways of solving the problem:
>
>   (1) Fix mod_dav, adjust mod_dav_svn accordingly
>
>   (2) Reimplement PROPFIND in mod_dav_svn
>
> Doing (1) is hard, since it requires revving mod_dav's API and special care
> for other modules using it.  As far as I know, those modules don't have tests
> and not breaking their behavior could be a challenging task.

(1) sounds like the right thing to do in the longer
term: fix the problem at the root and make everyone
profit from it henceforth.  But I have no idea of what
that entails and if your assessment is that it can't
be done easily - for various reasons - then I'll
believe that.

> I believe that currently we don't have the necessary resources (aka committers)
> to do that.  Given that the problem exists since 2003, I also doubt that it's
> going to solve itself in the nearby future.  And moreover, even if somehow
> managed to do that, the fix would probably only be available in the newer
> 2.6.x httpd releases.
>
> With that in mind, I propose we do (2).
>
> Although I'm not entirely happy with reimplementing a part of the protocol, I
> don't think that there are other realistic approaches to this serious problem.
> It wouldn't be the first time we're doing something like this, since we already
> handle POST ourselves, as well as provide the custom Subversion's HTTPv2
> protocol implementation.

Assuming that your analysis for (1) is correct,
I think (2) is a viable option.  However, I would
very much like to see mod_dav fixed as well and
to have (2) only as a temporary workaround.
"Temporary" being a quite euphemistic term here ...

> Apart from solving the memory usage issue, having the PROPFIND implementation
> in mod_dav_svn would allow us to improve a couple of other aspects of how we
> handle these requests, e.g.:
>
>   - Shrink down the PROPFIND response size by avoiding duplicate xmlns:
>     namespace declarations that currently exist.
>
>   - Optimize how we retrieve properties from the FS layer, since we would
>     no longer be doing mod_dav - mod_dav_svn - mod_dav transitions, and
>     would have more flexibility.
>
> I attached a proof-of-concept patch that does (2).  This patch solves the
> memory usage problem in my experiments and passes HTTPv2 / HTTPv1 tests.
 >
 > Thoughts?

Does the patch already include the improvements you
suggested above?  Is it feasible to #ifdef the usage
of that code?  I think if we have an easy and clean
opt-in / opt-out, people could be more accepting of
that workaround.

-- Stefan^2.

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Evgeny Kotkov wrote on Tue, Jan 05, 2016 at 19:08:41 +0300:
> Daniel Shahaf <d....@daniel.shahaf.name> writes:
> 
> >> I'm aware of two possible ways of solving the problem:
> >>
> >>  (1) Fix mod_dav, adjust mod_dav_svn accordingly
> >>
> >>  (2) Reimplement PROPFIND in mod_dav_svn
> >>
> 
> [...]
> 
> > I'm not too happy at basically saying I prefer (1) over (2) when you've
> > already written a 2000-line patch for (2)...  but maybe (2) _is_ the
> > better way to go and I just don't see it yet.  Enlighten me ☺
> 
> I did explore option (1) prior to (2), and spent a reasonable amount of time
> trying to prepare the mod_dav's part of the patch.
> 
> Option (1) requires a major overhauling of how mod_dav works with pools, and
> there are just too many things that we'd need to deal with.  Basically, every
> relevant pool usage needs to be examined, updated and propagated to the
> mod_dav's public API.
> 

I still do not see the problem.  mod_dav is not a huge amount of code,
and the work you describe does not sound too hard.  Why didn't your
attempt to patch mod_dav succeed?

> Then, even if we pulled that off in httpd, I don't think that these changes
> would become a part of 2.2.x and 2.4.x.  And the necessity of waiting or
> upgrading to httpd 2.6.x is a major setback, since I think that, for example,
> the 2.2.x series is still widely adopted.
> 

Why wouldn't they become part of 2.2.x?

Stefan's patch is a one-line bugfix patch that applies cleanly, so
anyone supporting httpd-2.2 can easily apply it.

If you're talking about a wider-scale refactoring, mod_dav has had
little change between 2.2.x and 2.6.x (mostly just the additional
argument for ap_log_error/dav_new_error), so backporting a mod_dav patch
to 2.2 would be... at worst, a lot of trivially-resolvable
conflicts; a SMOP, but not a reimplementation.

Even if such a wider-scale refactoring patch would be unfit in
upstream's opinion for an official 2.2.x patch release, it could be
maintained as a fork of httpd-2.2 that has the mod_dav-2.6 changes
backported to it; for example, it could live in
/repos/asf/httpd/httpd/branches/2.2.x-with-mod_dav-extensions, akin to
the javahl-1.8-extensions branch in our tree.

Either way, once the patch has been backported either to 2.2.x proper or
to 2.2.x-with-mod_dav-extensions, users can upgrade their mod_dav to
that (they won't even have to upgrade httpd proper, only mod_dav), and
we won't have to maintain a fork of mod_dav's code in our tree.

[ The above reads "2.2" but refers to both 2.2 and 2.4. ]

> Consuming unbounded amounts of memory or crashing is a serious issue,
> and people report it to us.  There's quite a history of plugging PROPFIND
> memory leaks in mod_dav_svn in response to such reports — see [1, 2], for
> instance.
> 
> [1] https://issues.apache.org/jira/browse/SVN-2000
> [2] https://svn.apache.org/r859602

Those two links are the same bug, which was fixed by having mod_dav_svn
clear a pool it allocated from.  How does this bug support your argument
that we should fork mod_dav's PROPFIND code in mod_dav_svn?

I do see that the only reason dav_resource_private::pool exists is that
mod_dav doesn't pass a scratch_pool argument its hooks.

> However, while the root cause is still there, the problem can be
> re-triggered by a variety of factors.  In this particular case, the trigger is
> the replacement of the dynamically-sized caches with a fixed-size membuffer
> cache in 1.7, because now series of cache misses or uncacheable objects can
> lead to performing allocations in the long-living request pool.

So you're saying a mod_dav backend can trigger suboptimal memory
allocation in mod_dav.  That should be fixed in mod_dav then, shouldn't
it?  Either by allocating memory in a better manner or by forbidding
backends from behaving in that way.

> Compared with (1), option (2) has known complexity and should fix the problem
> irrespectively of the httpd version, and that's why I proposed we do it.

Option (1) does not have "unknown complexity".  According to what you
wrote above, it involves converting mod_dav to dual pools.  That is
a finite task: for each of the 90 functions and 73 callbacks, at each
callsite, determine what lifetime the object it returns or constructs
needs, find a pool of that lifetime, and pass that pool to the function
or callback.

Option (1) also fixes the problem for users of all httpd versions.  It
has a higher bootstrap cost (backporting the patch to 2.2.x will involve
resolving trivial conflicts); however, it also has a lower ongoing
maintenance cost and a smaller chance of introducing bugs (because the
work would be done on httpd's list rather than on our lists).

I think your patch basically forks httpd's code, reindents it, and
improves it.  We should instead work with dev@httpd to seek a solution
that improves mod_dav in situ.

Daniel

P.S. Your patch probably shouldn't reindent the forked code from httpd's
indent style to svn's indent style, since that'll make merging bugfixes
between the fork and its upstream needlessly harder.


> 
> Regards,
> Evgeny Kotkov
> 

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Posted by Evgeny Kotkov <ev...@visualsvn.com>.
Daniel Shahaf <d....@daniel.shahaf.name> writes:

>> I'm aware of two possible ways of solving the problem:
>>
>>  (1) Fix mod_dav, adjust mod_dav_svn accordingly
>>
>>  (2) Reimplement PROPFIND in mod_dav_svn
>>

[...]

> Wait a minute.  Isn't "we have few active committers" a reason for
> choosing (1) over (2)?  I would naïvely expect "rewrite it" to be more
> expensive to implement/maintain than "patch it".
>
> Similarly, playing devil's advocate, if the issue exists since 2003,
> then presumably it's not urgent to fix, and waiting until httpd-2.6 is
> released might not be a major setback.

[...]

> I'm not too happy at basically saying I prefer (1) over (2) when you've
> already written a 2000-line patch for (2)...  but maybe (2) _is_ the
> better way to go and I just don't see it yet.  Enlighten me ☺

I did explore option (1) prior to (2), and spent a reasonable amount of time
trying to prepare the mod_dav's part of the patch.

Option (1) requires a major overhauling of how mod_dav works with pools, and
there are just too many things that we'd need to deal with.  Basically, every
relevant pool usage needs to be examined, updated and propagated to the
mod_dav's public API.

Then, even if we pulled that off in httpd, I don't think that these changes
would become a part of 2.2.x and 2.4.x.  And the necessity of waiting or
upgrading to httpd 2.6.x is a major setback, since I think that, for example,
the 2.2.x series is still widely adopted.

Consuming unbounded amounts of memory or crashing is a serious issue,
and people report it to us.  There's quite a history of plugging PROPFIND
memory leaks in mod_dav_svn in response to such reports — see [1, 2], for
instance.  However, while the root cause is still there, the problem can be
re-triggered by a variety of factors.  In this particular case, the trigger is
the replacement of the dynamically-sized caches with a fixed-size membuffer
cache in 1.7, because now series of cache misses or uncacheable objects can
lead to performing allocations in the long-living request pool.

Compared with (1), option (2) has known complexity and should fix the problem
irrespectively of the httpd version, and that's why I proposed we do it.

[1] https://issues.apache.org/jira/browse/SVN-2000
[2] https://svn.apache.org/r859602


Regards,
Evgeny Kotkov

Re: [RFC/PATCH] Handling PROPFIND in mod_dav_svn (was: Re: Issue with browsing a SVN 1.9.2, schema 7, packed, repository)

Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Evgeny Kotkov wrote on Wed, Dec 30, 2015 at 18:49:08 +0300:
> I'm aware of two possible ways of solving the problem:
> 
>  (1) Fix mod_dav, adjust mod_dav_svn accordingly
> 
>  (2) Reimplement PROPFIND in mod_dav_svn
> 
> Doing (1) is hard, since it requires revving mod_dav's API and special care
> for other modules using it.
>
> I believe that currently we don't have the necessary resources (aka committers)
> to do that.  Given that the problem exists since 2003, I also doubt that it's
> going to solve itself in the nearby future.  And moreover, even if somehow
> managed to do that, the fix would probably only be available in the newer
> 2.6.x httpd releases.
> 
> With that in mind, I propose we do (2).

Wait a minute.  Isn't "we have few active committers" a reason for
choosing (1) over (2)?  I would naïvely expect "rewrite it" to be more
expensive to implement/maintain than "patch it".

Similarly, playing devil's advocate, if the issue exists since 2003,
then presumably it's not urgent to fix, and waiting until httpd-2.6 is
released might not be a major setback.

("Is not going to solve itself" is avoidable rhetoric: of course it's
not going to fix itself, somebody is going to have to develop a mod_dav
patch.)

So these are three points in favour of (1) rather than (2).  What are
the reasons for (2) over (1)?

> Apart from solving the memory usage issue, having the PROPFIND implementation
> in mod_dav_svn would allow us to improve a couple of other aspects of how we
> handle these requests, e.g.:
> 
>  - Shrink down the PROPFIND response size by avoiding duplicate xmlns:
>    namespace declarations that currently exist.
> 
>  - Optimize how we retrieve properties from the FS layer, since we would
>    no longer be doing mod_dav - mod_dav_svn - mod_dav transitions, and
>    would have more flexibility.

Are these significant?  How much faster will a checkout/update be if
these improvements were implemented?  Would it be possible to implement
in httpd-2.6 mod_dav features that would achieve comparable performance
(when mod_dav_svn is run under httpd-2.6)?

> As far as I know, those modules don't have tests
> and not breaking their behavior could be a challenging task.
> 

You might want to double-check that with the httpd-dev crowd.  IIRC
httpd has some of its 'make test' functionality out-of-tree.

Or ask breser; I know he ran into this problem (changing
mod_dav/mod_dav_svn without breaking mod_dav_fs) and he might have found
a solution.

> Although I'm not entirely happy with reimplementing a part of the protocol, I
> don't think that there are other realistic approaches to this serious problem.
> It wouldn't be the first time we're doing something like this, since we already
> handle POST ourselves, as well as provide the custom Subversion's HTTPv2
> protocol implementation.
> 

Tangentially: We should find a new name for our POST-based protocol;
"HTTPv2" means something else to the wider world.

> I attached a proof-of-concept patch that does (2).  This patch solves the
> memory usage problem in my experiments and passes HTTPv2 / HTTPv1 tests.
> 
> Thoughts?

I'm not too happy at basically saying I prefer (1) over (2) when you've
already written a 2000-line patch for (2)...  but maybe (2) _is_ the
better way to go and I just don't see it yet.  Enlighten me ☺

Cheers,

Daniel