1.7.0-beta2 this afternoon

[Hyrum K Wright <hy...@hyrumwright.org>]

Since there seems to be consensus regarding not releasing 1.7.0-beta1,
I'll clean out STATUS and go roll beta2 this afternoon with the hopes
of releasing it by the end of the week.

I'm half-tempted to call it rc1 and start the soak.  Thoughts?

-Hyrum

Re: 1.7.0-beta2 this afternoon

[km...@rockwellcollins.com]

> From: Greg Stein <gs...@gmail.com>
> On Wed, Jul 20, 2011 at 09:23,  <km...@rockwellcollins.com> wrote:
> > Greg Stein <gs...@gmail.com> wrote on 07/19/2011 06:53:12 PM:
> >> On Tue, Jul 19, 2011 at 10:41, C. Michael Pilato 
<cm...@collab.net>
> >> wrote:
> >> >...
> >> > release-readiness?   What about ra_serf's "default-ness" -- is it 
in the
> >> > appropriate state based on that library's readiness at this moment?
> >>
> >> The only pending serf issue is my testing of a fix to memory blowup 
in
> >> ra_serf when somebody checks out a repository root (ie. 10's of
> >> thousands of files across trunk, tags, and branches). Personally, I
> >> think it is a bug that could be fixed in 1.7.1, but I'm shooting to
> >> get the code tested for a 1.7.0 release. It is currently disabled on
> >> trunk and 1.7.x, but a simple two-line fix enables it.
> >
> > I've seen revisions where millions (yes, I said millions) of files 
were
> > changed at one time, so I feel ra_serf needs to be able to handle
> > 10's of millions of files in trunk alone during an update without
> > needing TBs of memory...
> >
> > (I'm not saying I would recommend using Subversion in this way, just
> >  that I have seen older versions abused in amazing ways...)
> 
> Not saying it shouldn't be fixed... just saying that I'm unsure that
> it is a *release-blocker*. Or more precisely, that it is an issue that
> would cause serf to no longer be the default.

I guess I would consider a large increase in memory use a release
blocker/enough to make serf no longer the default.  In my experience,
10's of thousands of files is no longer a "large" test case.

Somewhat off-topic, but there was also previous concern that
the multiple connections that serf uses might overly stress some
larger servers.  Do we have any idea how many additional connections
a typical server would see?  For example, if I see 1000 concurrent
connections to a server with neon, will I need to support 10000 shorter
connections with serf?  (The 10x I chose is purely arbitrary and
not based upon any knowledge of the actual differences...)

> In any case, I've got all the code written, and a test plan and code
> for verification. It should make 1.7.0 without much problem (assuming
> that I haven't horrendously messed up my approach).

Good to hear we already know the potential solution!

Kevin R.

Re: 1.7.0-beta2 this afternoon

[Greg Stein <gs...@gmail.com>]

On Wed, Jul 20, 2011 at 09:23,  <km...@rockwellcollins.com> wrote:
> Greg Stein <gs...@gmail.com> wrote on 07/19/2011 06:53:12 PM:
>> On Tue, Jul 19, 2011 at 10:41, C. Michael Pilato <cm...@collab.net>
>> wrote:
>> >...
>> > release-readiness?   What about ra_serf's "default-ness" -- is it in the
>> > appropriate state based on that library's readiness at this moment?
>>
>> The only pending serf issue is my testing of a fix to memory blowup in
>> ra_serf when somebody checks out a repository root (ie. 10's of
>> thousands of files across trunk, tags, and branches). Personally, I
>> think it is a bug that could be fixed in 1.7.1, but I'm shooting to
>> get the code tested for a 1.7.0 release. It is currently disabled on
>> trunk and 1.7.x, but a simple two-line fix enables it.
>
> I've seen revisions where millions (yes, I said millions) of files were
> changed at one time, so I feel ra_serf needs to be able to handle
> 10's of millions of files in trunk alone during an update without
> needing TBs of memory...
>
> (I'm not saying I would recommend using Subversion in this way, just
>  that I have seen older versions abused in amazing ways...)

Not saying it shouldn't be fixed... just saying that I'm unsure that
it is a *release-blocker*. Or more precisely, that it is an issue that
would cause serf to no longer be the default.

In any case, I've got all the code written, and a test plan and code
for verification. It should make 1.7.0 without much problem (assuming
that I haven't horrendously messed up my approach).

Cheers,
-g

Re: 1.7.0-beta2 this afternoon

[km...@rockwellcollins.com]

Greg Stein <gs...@gmail.com> wrote on 07/19/2011 06:53:12 PM:
> On Tue, Jul 19, 2011 at 10:41, C. Michael Pilato <cm...@collab.net> 
wrote:
> >...
> > release-readiness?   What about ra_serf's "default-ness" -- is it in 
the
> > appropriate state based on that library's readiness at this moment?
> 
> The only pending serf issue is my testing of a fix to memory blowup in
> ra_serf when somebody checks out a repository root (ie. 10's of
> thousands of files across trunk, tags, and branches). Personally, I
> think it is a bug that could be fixed in 1.7.1, but I'm shooting to
> get the code tested for a 1.7.0 release. It is currently disabled on
> trunk and 1.7.x, but a simple two-line fix enables it.

I've seen revisions where millions (yes, I said millions) of files were
changed at one time, so I feel ra_serf needs to be able to handle
10's of millions of files in trunk alone during an update without
needing TBs of memory...

(I'm not saying I would recommend using Subversion in this way, just
 that I have seen older versions abused in amazing ways...)

Kevin R.

Re: 1.7.0-beta2 this afternoon

[Greg Stein <gs...@gmail.com>]

On Thu, Jul 21, 2011 at 08:09, Philip Martin <ph...@wandisco.com> wrote:
> Greg Stein <gs...@gmail.com> writes:
>
>>> The first thing I tried leaks memory:
>>> http://subversion.tigris.org/issues/show_bug.cgi?id=3967
>>
>> Are you suggesting that my fix for 3888 caused this problem, or is
>> incorrect in some way? Or is this a new, distinct, and undiscovered
>> problem that you're bringing up?
>
> Issue 3888 was something to do with file transfers?  I see 3967 with
> empty revisions so it's probably different.

The root problem with 3888 is that N server requests would be queued
up, where N could grow into the tens of thousands. There are various
bits to try and minimize the memory space of each of those requests,
but it still ended up needing a huge/unbounded amount of memory.

The fix was to halt processing of the server response (that indicated
*which* requests to make) when the outstanding request count gets too
high. Once the count drops, then response processing resumes and more
requests are queued.

If the svnrdump process generates a hojillion independent requests,
then it may be seeing a similar problem. Applying throttling similar
to the update process should fix it.

If the process is single-request based (eg. 'svn log' and a million
log entries in a stream), then we'd have to look elsewhere. That would
be an iterpool-like symptom such as we just found in the log
processing.

Cheers,
-g

Re: 1.7.0-beta2 this afternoon

[Philip Martin <ph...@wandisco.com>]

Greg Stein <gs...@gmail.com> writes:

>> The first thing I tried leaks memory:
>> http://subversion.tigris.org/issues/show_bug.cgi?id=3967
>
> Are you suggesting that my fix for 3888 caused this problem, or is
> incorrect in some way? Or is this a new, distinct, and undiscovered
> problem that you're bringing up?

Issue 3888 was something to do with file transfers?  I see 3967 with
empty revisions so it's probably different.

-- 
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

Re: 1.7.0-beta2 this afternoon

[Greg Stein <gs...@gmail.com>]

On Thu, Jul 21, 2011 at 06:44, Philip Martin <ph...@wandisco.com> wrote:
> Greg Stein <gs...@gmail.com> writes:
>...
>> Yup. It was listed as an issue for the 1.7.0 release. Now that it has
>> been fixed.. sure: it should be merged to 1.7.x, and we're good to go.
>
> Enable a whole bunch of new code, and then say "look no bug reported" :)

Not sure what you're asking for here. We had issue 3888 for fixing in
1.7.0. It has been fixed. I have tested the change *extensively*, and
am aware of no bugs at all in that code. We wanted the fix in 1.7.0,
so it ought to be merged.

Now... you're saying something separate below:

> The first thing I tried leaks memory:
> http://subversion.tigris.org/issues/show_bug.cgi?id=3967

Are you suggesting that my fix for 3888 caused this problem, or is
incorrect in some way? Or is this a new, distinct, and undiscovered
problem that you're bringing up?

Cheers,
-g

Re: 1.7.0-beta2 this afternoon

[Philip Martin <ph...@wandisco.com>]

Greg Stein <gs...@gmail.com> writes:

> On Thu, Jul 21, 2011 at 05:14, Justin Erenkrantz <ju...@erenkrantz.com> wrote:
>> On Thu, Jul 21, 2011 at 7:09 AM, Greg Stein <gs...@gmail.com> wrote:
>>> ... and tested and fixed. See the 1.7.x-issue3888 branch. It is
>>> nominated for backport.
>>
>> I just reviewed all of the related commits (including what was
>> committed before the branch) - it looks good here and I added my +1 to
>> 1.7.x/STATUS.  Lots of cute-ness...but...nice.  =)
>>
>> I believe this is the last blocker that has been raised related to
>> ra_serf...right?  -- justin
>
> Yup. It was listed as an issue for the 1.7.0 release. Now that it has
> been fixed.. sure: it should be merged to 1.7.x, and we're good to go.

Enable a whole bunch of new code, and then say "look no bug reported" :)

The first thing I tried leaks memory:
http://subversion.tigris.org/issues/show_bug.cgi?id=3967

-- 
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

Re: 1.7.0-beta2 this afternoon

[Greg Stein <gs...@gmail.com>]

On Thu, Jul 21, 2011 at 05:14, Justin Erenkrantz <ju...@erenkrantz.com> wrote:
> On Thu, Jul 21, 2011 at 7:09 AM, Greg Stein <gs...@gmail.com> wrote:
>> ... and tested and fixed. See the 1.7.x-issue3888 branch. It is
>> nominated for backport.
>
> I just reviewed all of the related commits (including what was
> committed before the branch) - it looks good here and I added my +1 to
> 1.7.x/STATUS.  Lots of cute-ness...but...nice.  =)
>
> I believe this is the last blocker that has been raised related to
> ra_serf...right?  -- justin

Yup. It was listed as an issue for the 1.7.0 release. Now that it has
been fixed.. sure: it should be merged to 1.7.x, and we're good to go.

We have no known release blockers, though we've been churning the
backporting. Thus, the idea is to release a beta2, and "soon" to move
to a release candidate. I think an RC is more about waiting for the
churn to soften.

Cheers,
-g

Re: 1.7.0-beta2 this afternoon

[Justin Erenkrantz <ju...@erenkrantz.com>]

On Thu, Jul 21, 2011 at 7:09 AM, Greg Stein <gs...@gmail.com> wrote:
> ... and tested and fixed. See the 1.7.x-issue3888 branch. It is
> nominated for backport.

I just reviewed all of the related commits (including what was
committed before the branch) - it looks good here and I added my +1 to
1.7.x/STATUS.  Lots of cute-ness...but...nice.  =)

I believe this is the last blocker that has been raised related to
ra_serf...right?  -- justin

Re: 1.7.0-beta2 this afternoon

[Greg Stein <gs...@gmail.com>]

On Tue, Jul 19, 2011 at 19:53, Greg Stein <gs...@gmail.com> wrote:
> On Tue, Jul 19, 2011 at 10:41, C. Michael Pilato <cm...@collab.net> wrote:
>>...
>> release-readiness?   What about ra_serf's "default-ness" -- is it in the
>> appropriate state based on that library's readiness at this moment?
>
> The only pending serf issue is my testing of a fix to memory blowup in
> ra_serf when somebody checks out a repository root (ie. 10's of

... and tested and fixed. See the 1.7.x-issue3888 branch. It is
nominated for backport.

Cheers,
-g

Re: 1.7.0-beta2 this afternoon

[Greg Stein <gs...@gmail.com>]

On Tue, Jul 19, 2011 at 19:53, Greg Stein <gs...@gmail.com> wrote:
>...
> The only pending serf issue is my testing of a fix to memory blowup in
> ra_serf when somebody checks out a repository root (ie. 10's of

There appears to be a bug in my debug code, or the original code. The
night is done, but I will continue this tomorrow.

I've committed the testing code, along with a bunch of prose to
explain the approach (see notes/ra-serf-testing.txt). Now it is just
tweaking.

Cheers,
-g

Re: 1.7.0-beta2 this afternoon

[Greg Stein <gs...@gmail.com>]

On Tue, Jul 19, 2011 at 10:41, C. Michael Pilato <cm...@collab.net> wrote:
>...
> release-readiness?   What about ra_serf's "default-ness" -- is it in the
> appropriate state based on that library's readiness at this moment?

The only pending serf issue is my testing of a fix to memory blowup in
ra_serf when somebody checks out a repository root (ie. 10's of
thousands of files across trunk, tags, and branches). Personally, I
think it is a bug that could be fixed in 1.7.1, but I'm shooting to
get the code tested for a 1.7.0 release. It is currently disabled on
trunk and 1.7.x, but a simple two-line fix enables it.

Cheers,
-g

Re: 1.7.0-beta2 this afternoon

[Justin Erenkrantz <ju...@erenkrantz.com>]

On Wed, Jul 20, 2011 at 6:33 AM, Hyrum K Wright <hy...@hyrumwright.org> wrote:
> And I citing Mike's concerns about about RC-ness, I'm inclined to make
> this one beta2.

+1.  -- justin

Re: 1.7.0-beta2 this afternoon

[Hyrum K Wright <hy...@hyrumwright.org>]

On Tue, Jul 19, 2011 at 9:41 AM, C. Michael Pilato <cm...@collab.net> wrote:
> On 07/19/2011 10:15 AM, Hyrum K Wright wrote:
>> Since there seems to be consensus regarding not releasing 1.7.0-beta1,
>> I'll clean out STATUS and go roll beta2 this afternoon with the hopes
>> of releasing it by the end of the week.
>>
>> I'm half-tempted to call it rc1 and start the soak.  Thoughts?
>
> I don't follow the arguments for not releasing beta1, but I'm late to the
> party and accept the consequences thereof.  Still, you can't call it an RC
> if it's not release-ready.
>
> I know that there are some changes that we allow between an RC and final
> release without restarting the soak (updating CHANGES, for example).  But
> there are still several yellow in-progress items in roadmap.html.  Most of
> them I could probably assign a committer's name to as the person who seemed
> to be carrying the banner for that task.  Is anyone able/willing to review
> and update the status of those items so we can enter RC1 with a sense of
> release-readiness?   What about ra_serf's "default-ness" -- is it in the
> appropriate state based on that library's readiness at this moment?

After some discussion on IRC, Greg and others suggested I post
tarballs on Thursday afternoon, to allow folks time to kick off their
tests over the weekend and an early-week release next week.  I'm
inclined to go with that plan, barring any objections.

And I citing Mike's concerns about about RC-ness, I'm inclined to make
this one beta2.

-Hyrum

Re: 1.7.0-beta2 this afternoon

["C. Michael Pilato" <cm...@collab.net>]

On 07/19/2011 10:15 AM, Hyrum K Wright wrote:
> Since there seems to be consensus regarding not releasing 1.7.0-beta1,
> I'll clean out STATUS and go roll beta2 this afternoon with the hopes
> of releasing it by the end of the week.
> 
> I'm half-tempted to call it rc1 and start the soak.  Thoughts?

I don't follow the arguments for not releasing beta1, but I'm late to the
party and accept the consequences thereof.  Still, you can't call it an RC
if it's not release-ready.

I know that there are some changes that we allow between an RC and final
release without restarting the soak (updating CHANGES, for example).  But
there are still several yellow in-progress items in roadmap.html.  Most of
them I could probably assign a committer's name to as the person who seemed
to be carrying the banner for that task.  Is anyone able/willing to review
and update the status of those items so we can enter RC1 with a sense of
release-readiness?   What about ra_serf's "default-ness" -- is it in the
appropriate state based on that library's readiness at this moment?

-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Distributed Development On Demand