You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by David Summers <da...@summersoft.fay.ar.us> on 2007/10/23 03:14:37 UTC
Re: New (1.5) feature questions -- things we need to address before
a release can happen
A thread got started this last May about "things we need to address before
a release can happen."
One of those is something that I run into sporadically (happened again
today).
There was discussion about it at
http://svn.haxx.se/dev/archive-2007-05/0404.shtml
but then the thread died.
When a large commit happens to the slave server (re-directed to the master
via web-dav proxy) then sometimes for a long time (1.5 hours this last
time) the user got errors on his client (I can get the exact error if
people are interested) until the slave synced up with the master
repository.
This happens 100% of the time if the commit is beyond some magic number of
megabytes (or seconds, not sure which).
Should I enter this as a problem in the issue tracker so we don't forget
about it?
Karl had some suggestions about a couple of different ways to fix it.
Is this something that would be "easy" to do?
It would be SUPER SWEET if this could be fixed for 1.5.0.
If it is "easy" (how easy?) then is it "byte-size" or something that
someone not very familiar with the Subversion code (me?) could do?
At the time I mentioned it this last May I was extremely busy but at the
moment I have more time (still busy, can't promise anything though).
The other problem I've had with master/slave web-dav proxy mirroring is
that it *should* still be able to read the slave even when the master is
unreachable, but this turns out not to be the case. I had a discussion at
http://svn.haxx.se/dev/archive-2007-03/0823.shtml and I sent a tcpdump but
never got a reply.
Should I enter this as an issue as well? This is only a problem when the
master server is down (thankfully rare) but I'm also wondering if extra
traffic is being sent to the master because of this?
Thanks!
--
David Wayne Summers "Linux: Because reboots are for hardware upgrades!"
david@summersoft.fay.ar.us PGP Key: http://summersoft.fay.ar.us/~david/pgp.txt
PGP Key fingerprint = 0B44 B118 85CC F4EC 7021 1ED4 1516 5B78 E320 2001
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by David Summers <da...@summersoft.fay.ar.us>.
On Thu, 25 Oct 2007, Jack Repenning wrote:
> On Oct 25, 2007, at 6:41 AM, David Summers wrote:
>
>
>> The master is in Huntsville, AL, and the slave is in Lowell, AR, across the
>> Internet. Most of the time there is no problem at all but when the the
>> commit passes some magic number of bytes or seconds then the client freaks
>> out. The other day, I kept getting errors from the client for 1.5 hours
>> after the (big) commit until the slave and master had synced up.
>
> Does the commit part of the commit take 1.5 hours? That is, does it seem
> like this commit is big enough that "1.5 hours" is a reasonable estimate for
> how long it takes to transfer the data?
>
> We've all been kind of assuming that this 1.5 hours was the legitimate time
> needed for a post-commit svnsync to transfer the data back to the slave(s).
> But, geez, 1.5 hours? On a one time-zone US Internet connection? What are
> we committing, here, terabytes? Maybe the explanation's a bit deeper,
> somehow?
>
I believe that the original commit took approx as long as the sync, so
yes, it probably took about 1 - 1.5 hours for the commit. It was probably
on the order of 1 Gigabyte across the internet for the original commit and
I'm fairly sure it was about 1.5 hours for the svnsync back to our local
slave server from the master in Huntsville because I started watching the
network traffic between the machines while I compared the repository
versions on both machines by browsing them in my web browser.
--
David Wayne Summers "Linux: Because reboots are for hardware upgrades!"
david@summersoft.fay.ar.us PGP Key: http://summersoft.fay.ar.us/~david/pgp.txt
PGP Key fingerprint = 0B44 B118 85CC F4EC 7021 1ED4 1516 5B78 E320 2001
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before a release can happen
Posted by Jack Repenning <jr...@collab.net>.
On Oct 25, 2007, at 6:41 AM, David Summers wrote:
> The master is in Huntsville, AL, and the slave is in Lowell, AR,
> across the Internet. Most of the time there is no problem at all
> but when the the commit passes some magic number of bytes or
> seconds then the client freaks out. The other day, I kept getting
> errors from the client for 1.5 hours after the (big) commit until
> the slave and master had synced up.
Does the commit part of the commit take 1.5 hours? That is, does it
seem like this commit is big enough that "1.5 hours" is a reasonable
estimate for how long it takes to transfer the data?
We've all been kind of assuming that this 1.5 hours was the
legitimate time needed for a post-commit svnsync to transfer the data
back to the slave(s). But, geez, 1.5 hours? On a one time-zone US
Internet connection? What are we committing, here, terabytes? Maybe
the explanation's a bit deeper, somehow?
-==-
Jack Repenning
Chief Technology Officer
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
mobile: +1 408.835.8090
raindance: +1 877.326.2337, x844.7461
aim: jackrepenning
skype: jrepenning
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by David Summers <da...@summersoft.fay.ar.us>.
On Wed, 24 Oct 2007, Justin Erenkrantz wrote:
> On Oct 22, 2007 8:14 PM, David Summers <da...@summersoft.fay.ar.us> wrote:
>> When a large commit happens to the slave server (re-directed to the master
>> via web-dav proxy) then sometimes for a long time (1.5 hours this last
>> time) the user got errors on his client (I can get the exact error if
>> people are interested) until the slave synced up with the master
>> repository.
>
> Hmm - it sounds like the svnsync isn't executing. Is this the case?
> svnsync *should* be relatively fast - especially if the link between
> slave<->master is fast enough.
I have each commit start a svnsync from the master to the slave server.
> But, I don't think this is a solvable problem - Karl's solution would
> require a lot more intelligence in the proxy than there is now. You
> just have to rely upon the synchronization happening fast enough. --
> justin
>
The master is in Huntsville, AL, and the slave is in Lowell, AR, across
the Internet. Most of the time there is no problem at all but when the
the commit passes some magic number of bytes or seconds then the client
freaks out. The other day, I kept getting errors from the client for 1.5
hours after the (big) commit until the slave and master had synced up.
Sometimes we have to mess around with the client files to get it to stop
misbehaving (waving hands here, I don't recall all the things I've tried).
David Wayne Summers "Linux: Because reboots are for hardware upgrades!"
david@summersoft.fay.ar.us PGP Key: http://summersoft.fay.ar.us/~david/pgp.txt
PGP Key fingerprint = 0B44 B118 85CC F4EC 7021 1ED4 1516 5B78 E320 2001
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before a release can happen
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Oct 22, 2007 8:14 PM, David Summers <da...@summersoft.fay.ar.us> wrote:
> When a large commit happens to the slave server (re-directed to the master
> via web-dav proxy) then sometimes for a long time (1.5 hours this last
> time) the user got errors on his client (I can get the exact error if
> people are interested) until the slave synced up with the master
> repository.
Hmm - it sounds like the svnsync isn't executing. Is this the case?
svnsync *should* be relatively fast - especially if the link between
slave<->master is fast enough.
But, I don't think this is a solvable problem - Karl's solution would
require a lot more intelligence in the proxy than there is now. You
just have to rely upon the synchronization happening fast enough. --
justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by David Summers <da...@summersoft.fay.ar.us>.
On Mon, 22 Oct 2007, Karl Fogel wrote:
> David Summers <da...@summersoft.fay.ar.us> writes:
>> When a large commit happens to the slave server (re-directed to the
>> master via web-dav proxy) then sometimes for a long time (1.5 hours
>> this last time) the user got errors on his client (I can get the exact
>> error if people are interested) until the slave synced up with the
>> master repository.
>>
>> This happens 100% of the time if the commit is beyond some magic
>> number of megabytes (or seconds, not sure which).
>>
>> Should I enter this as a problem in the issue tracker so we don't
>> forget about it?
>
> +1
Posted as issue # 2988.
>> Karl had some suggestions about a couple of different ways to fix it.
>>
>> Is this something that would be "easy" to do?
>
> I don't think so. The right fix is for the slave to proxy over to the
> master for reads as well, when a client requests a revision higher
> than the slave's current youngest.
>
> An alternative (for less load on the slave) is for the client to
> notice the particular error from the slave, and contact the master
> itself. But that might have working copy metadata implications.
>
> I don't think that...
>
>> It would be SUPER SWEET if this could be fixed for 1.5.0.
>
> ... solving this for 1.5.0 is realistic, though would love to be
> proven wrong of course.
>
David Wayne Summers "Linux: Because reboots are for hardware upgrades!"
david@summersoft.fay.ar.us PGP Key: http://summersoft.fay.ar.us/~david/pgp.txt
PGP Key fingerprint = 0B44 B118 85CC F4EC 7021 1ED4 1516 5B78 E320 2001
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by Jay Levitt <li...@shopwatch.org>.
The original message was received at Tue, 23 Oct 2007 07:26:11 -0400
from office.home.jay.fm [192.168.1.151]
----- The following addresses had permanent fatal errors -----
<de...@subversion.tigris.org>
(reason: Can't create output)
----- Transcript of session follows -----
procmail: [2596] Tue Oct 23 07:26:16 2007
procmail: Assigning "LOGFILE=/var/log/procmail"
procmail: Opening "/var/log/procmail"
550 5.0.0 <de...@subversion.tigris.org>... Can't create output
Reporting-MTA: dns; server.home.jay.fm
Received-From-MTA: DNS; office.home.jay.fm
Arrival-Date: Tue, 23 Oct 2007 07:26:11 -0400
Final-Recipient: RFC822; dev@subversion.tigris.org
Action: failed
Status: 5.3.0
Diagnostic-Code: X-Unix; 73
Last-Attempt-Date: Tue, 23 Oct 2007 07:26:16 -0400
Subject:
Re: New (1.5) feature questions -- things we need to address before a
release can happen
From:
Jay Levitt <li...@shopwatch.org>
Date:
Tue, 23 Oct 2007 07:25:59 -0400
To:
Jack Repenning <jr...@collab.net>
CC:
Karl Fogel <kf...@red-bean.com>, David Summers
<da...@summersoft.fay.ar.us>,
dev@subversion.tigris.org
On 10/23/2007 2:25 AM, Jack Repenning wrote:
> On Oct 23, 2007, at 11:38 AM, Karl Fogel wrote:
>
>>> Karl had some suggestions about a couple of different ways to fix it.
>>>
>>> Is this something that would be "easy" to do?
>>
>> ...
>> An alternative (for less load on the slave) is for the client to
>> notice the particular error from the slave, and contact the master
>> itself. But that might have working copy metadata implications.
>
> Or even network topology challenges. It would be easy to set up a
configuration where the clients are firewalled away from the master, can
only speak to the slave (and only the slave can speak to the master).
I'm not sure I believe it would be a good idea: basically you're putting
your server into the network DMZ, but your server's the data you want
best guarded, not least! But just 'cause I don't think it's a useful
idea does not mean some crazy IT manager won't do it. Indeed, simply
because it's called a "Proxy" might make this more likely, as that word
is also widely used for "the computer in the DMZ that the clients talk
to in order to get to the Internet."
Actually, that probably won't be all that uncommon, if you think of a
company with multiple data centers, instead of "inside/outside". I can
imagine a network where your west-coast and east-coast data centers can
talk to each other, but your west-coast office can only talk to the
west-coast data center, and the east-coast office can only talk to the
east-coast data center.
Jay Levitt
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before a release can happen
Posted by Jack Repenning <jr...@collab.net>.
On Oct 23, 2007, at 11:38 AM, Karl Fogel wrote:
>> Karl had some suggestions about a couple of different ways to fix it.
>>
>> Is this something that would be "easy" to do?
>
> ...
> An alternative (for less load on the slave) is for the client to
> notice the particular error from the slave, and contact the master
> itself. But that might have working copy metadata implications.
Or even network topology challenges. It would be easy to set up a
configuration where the clients are firewalled away from the master,
can only speak to the slave (and only the slave can speak to the
master). I'm not sure I believe it would be a good idea: basically
you're putting your server into the network DMZ, but your server's
the data you want best guarded, not least! But just 'cause I don't
think it's a useful idea does not mean some crazy IT manager won't do
it. Indeed, simply because it's called a "Proxy" might make this
more likely, as that word is also widely used for "the computer in
the DMZ that the clients talk to in order to get to the Internet."
-==-
Jack Repenning
Chief Technology Officer
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
mobile: +1 408.835.8090
raindance: +1 877.326.2337, x844.7461
aim: jackrepenning
skype: jrepenning
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by David Summers <da...@summersoft.fay.ar.us>.
On Wed, 24 Oct 2007, Justin Erenkrantz wrote:
> On Oct 23, 2007 7:37 PM, David Summers <da...@summersoft.fay.ar.us> wrote:
>>>> Should I enter this as an issue as well? This is only a problem when
>>>> the master server is down (thankfully rare) but I'm also wondering if
>>>> extra traffic is being sent to the master because of this?
>>>
>>> Huh, yeah, that sounds like a separate bug, but still a bug.
>>>
>>> Thanks for not forgetting about these issues!
>>>
>>
>> Posted as issue # 2989.
>
> I think this was resolved with REPORT being handled locally. Please
> try it again with trunk, but I have a hunch this might already be
> fixed. -- justin
>
Cool! Will try tomorrow when I get to work and will report back.
Thanks!
--
David Wayne Summers "Linux: Because reboots are for hardware upgrades!"
david@summersoft.fay.ar.us PGP Key: http://summersoft.fay.ar.us/~david/pgp.txt
PGP Key fingerprint = 0B44 B118 85CC F4EC 7021 1ED4 1516 5B78 E320 2001
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before a release can happen
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Oct 23, 2007 7:37 PM, David Summers <da...@summersoft.fay.ar.us> wrote:
> >> Should I enter this as an issue as well? This is only a problem when
> >> the master server is down (thankfully rare) but I'm also wondering if
> >> extra traffic is being sent to the master because of this?
> >
> > Huh, yeah, that sounds like a separate bug, but still a bug.
> >
> > Thanks for not forgetting about these issues!
> >
>
> Posted as issue # 2989.
I think this was resolved with REPORT being handled locally. Please
try it again with trunk, but I have a hunch this might already be
fixed. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by David Summers <da...@summersoft.fay.ar.us>.
On Mon, 22 Oct 2007, Karl Fogel wrote:
> David Summers <da...@summersoft.fay.ar.us> writes:
>> The other problem I've had with master/slave web-dav proxy mirroring
>> is that it *should* still be able to read the slave even when the
>> master is unreachable, but this turns out not to be the case. I had a
>> discussion at
>> http://svn.haxx.se/dev/archive-2007-03/0823.shtml and I sent a tcpdump
>> but never got a reply.
>>
>> Should I enter this as an issue as well? This is only a problem when
>> the master server is down (thankfully rare) but I'm also wondering if
>> extra traffic is being sent to the master because of this?
>
> Huh, yeah, that sounds like a separate bug, but still a bug.
>
> Thanks for not forgetting about these issues!
>
Posted as issue # 2989.
David Wayne Summers "Linux: Because reboots are for hardware upgrades!"
david@summersoft.fay.ar.us PGP Key: http://summersoft.fay.ar.us/~david/pgp.txt
PGP Key fingerprint = 0B44 B118 85CC F4EC 7021 1ED4 1516 5B78 E320 2001
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before a release can happen
Posted by Karl Fogel <kf...@red-bean.com>.
David Summers <da...@summersoft.fay.ar.us> writes:
> When a large commit happens to the slave server (re-directed to the
> master via web-dav proxy) then sometimes for a long time (1.5 hours
> this last time) the user got errors on his client (I can get the exact
> error if people are interested) until the slave synced up with the
> master repository.
>
> This happens 100% of the time if the commit is beyond some magic
> number of megabytes (or seconds, not sure which).
>
> Should I enter this as a problem in the issue tracker so we don't
> forget about it?
+1
> Karl had some suggestions about a couple of different ways to fix it.
>
> Is this something that would be "easy" to do?
I don't think so. The right fix is for the slave to proxy over to the
master for reads as well, when a client requests a revision higher
than the slave's current youngest.
An alternative (for less load on the slave) is for the client to
notice the particular error from the slave, and contact the master
itself. But that might have working copy metadata implications.
I don't think that...
> It would be SUPER SWEET if this could be fixed for 1.5.0.
... solving this for 1.5.0 is realistic, though would love to be
proven wrong of course.
> The other problem I've had with master/slave web-dav proxy mirroring
> is that it *should* still be able to read the slave even when the
> master is unreachable, but this turns out not to be the case. I had a
> discussion at
> http://svn.haxx.se/dev/archive-2007-03/0823.shtml and I sent a tcpdump
> but never got a reply.
>
> Should I enter this as an issue as well? This is only a problem when
> the master server is down (thankfully rare) but I'm also wondering if
> extra traffic is being sent to the master because of this?
Huh, yeah, that sounds like a separate bug, but still a bug.
Thanks for not forgetting about these issues!
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by Lieven Govaerts <sv...@mobsol.be>.
Blair Zajac wrote:
> David Summers wrote:
>>
>> A thread got started this last May about "things we need to address
>> before a release can happen."
>>
>> One of those is something that I run into sporadically (happened again
>> today).
>>
>> There was discussion about it at
>> http://svn.haxx.se/dev/archive-2007-05/0404.shtml
>> but then the thread died.
>>
>>
>> When a large commit happens to the slave server (re-directed to the
>> master via web-dav proxy) then sometimes for a long time (1.5 hours
>> this last time) the user got errors on his client (I can get the exact
>> error if people are interested) until the slave synced up with the
>> master repository.
>
> In this case, don't you want to have the post-commit script block until
> it brings all the mirrors up to date with svnsync? This is in contrast
> to the normal situation where redirect std* to get their background
> processes to detach from post-commit.
>
You don't want to wait until all slaves are brought up to date, but only
for the sync to the slave that the client has used for the commit.
I suggested last week at SubConf to let the slave initiate the sync when
it finds out the commit has finished. I don't know how easy it is to
implement, but in concept this looks like the better model.
Lieven
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by Blair Zajac <bl...@orcaware.com>.
David Summers wrote:
>
> A thread got started this last May about "things we need to address
> before a release can happen."
>
> One of those is something that I run into sporadically (happened again
> today).
>
> There was discussion about it at
> http://svn.haxx.se/dev/archive-2007-05/0404.shtml
> but then the thread died.
>
>
> When a large commit happens to the slave server (re-directed to the
> master via web-dav proxy) then sometimes for a long time (1.5 hours this
> last time) the user got errors on his client (I can get the exact error
> if people are interested) until the slave synced up with the master
> repository.
In this case, don't you want to have the post-commit script block until it
brings all the mirrors up to date with svnsync? This is in contrast to the
normal situation where redirect std* to get their background processes to detach
from post-commit.
Blair
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org