You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Mark Phippard <ma...@gmail.com> on 2007/05/20 15:53:14 UTC
New (1.5) feature questions -- things we need to address before a release can happen
I am doing a webinar at the end of the month and was going to do a few
slides on some of the new features in 1.5. We are going to need this
information to update the release note as well though.
It seems like there are a few new features that are really lacking in
documentation and information even though they have been in trunk for
a long time. Perhaps more significantly, they seem like fairly useful
features and thus deserve more documentation and attention.
SASL
Where is the notes file that explains the feature? What can you do
with it? Does it, or does it not support TLS or SSL? Does it work on
Windows? What is involved in bulding support for it into the client
and server? What happens if client and server do not both have the
support compiled in? How does one set it up etc. Given that we are
reasonably close to a 1.5 release this information should all be
readily available.
DAV write-through proxy
This has the potential to be a killer feature, but it is another one
that is not heavily documented. There is a notes file, but it looks
like it pre-dates the implementation in some of the details. This one
looks fairly easy to turn on but could benefit from some more
explanation about what it wants to accomplish with some real-world
implementation scenarios. For example, it seems fairly important that
the read-only "slaves" are updated after a commit. Presumably svnsync
would be a good candidate for doing this. An example hook-based
implementation would be useful. Also, some discussion on the
ramifications would be useful. What if I do a commit, followed by an
update and my mirror has not been updated yet? Likewise, what if
another user does a commit, and then I do a commit by mirror has not
been updated yet. Will that effect me?
I do not know any of these answers, or I would not be asking. I am
willing to help collect and format the information and get it into the
right places. I have tried searching mailing list threads but that
has often just led to more questions as I see references to things
like "I need to add TLS support and will post something ..." then I
cannot find that it was ever done.
It would be unlike this project to include features like this with so
little documentation available. By the way, although merge tracking
has a lot of documentation, one area that has not been well addressed
is the new SQLite dependency and its impact on the build process. In
particular, building on Windows is more of a black art than ever.
If it would be helpful, I can lead/organize the effort on fixing some
of these deficiencies, but it will really take "subject matter
experts" on these items to step forward and provide the information
and make sure the final output is accurate and complete.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before a release can happen
Posted by Karl Fogel <kf...@red-bean.com>.
"Mark Phippard" <ma...@gmail.com> writes:
> It seems like if the server can seamlessly re-direct a commit to the
> master server, it could potentially do the same for an update, when it
> detected the need.
>
> That being said, doing so might cover up that there is a problem with
> the slave server that no one realizes exists. It seems like if the
> client got an error, but was no worse for it once the slave was synced
> would be the ideal.
That would certainly be the easiest thing to do; it might be a good
way to start, then we can see how annoying it is and make a more
sophisticated behavior if necessary.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before a release can happen
Posted by Mark Phippard <ma...@gmail.com>.
On 5/20/07, Karl Fogel <kf...@red-bean.com> wrote:
> "Mark Phippard" <ma...@gmail.com> writes:
> >> Generally it works well, however when the commit is "large" (I'm not sure,
> >> several megabytes or more or at least the commit time of a few minutes
> >> (???)) then we usually have problems where the slave has not synced up
> >> with
> >> the master yet and the client that committed it to the slave gets confused
> >> and starts giving errors. It has been possible but problematic to get
> >> things straightened out, and sometimes I've had a svnsync not go through
> >> and had to manually kick it and/or delete the svn:lock (I think that is
> >> the property) to get things going again.
> >>
> >> Is there a way to do a synchronous sync with the slave or some code magic
> >> behind the scenes that could be added in the client or slave server or
> >> master server to tell the client to wait a little longer to be synced
> >> up with the slave server correctly?
> >>
> >> I think what is happening is that the slave says "I've committed a certain
> >> version" and then at the end of the commit it checks that the slave server
> >> has that version which hasn't been synced yet and it gets confused.
> >>
> >> Thankfully this has not happened a whole lot but it has happened on
> >> multiple occasions. I'm telling my users that when they have a "large"
> >> commit (size or time is hard to determine) then to switch to the master to
> >> commit it and then switch back to the slave again.
> >>
> >> Any help, ideas, pointers, would be appreciated.
> >
> > Seems like a problem that should be identified and fixed before we
> > release. Now that we have svnsync it would be super-cool if it could
> > be formally integrated into this so that you did not need a hook at
> > all.
>
> A client-side solution might work. Take a client that's reading from
> a slave repository S, but whose commits go to a master M.
>
> 1. S is at revision R, M is at revision R
>
> 2. Now the client commits. Afterwards, M is at revision R+1, but S
> is still at revision R. (Assume the commit is large, so S can't
> sync up immediately.)
>
> 3. If the client knows that its commit went through to M, it can
> obey a rule that says: "Until S is at R+1 or higher, do all
> reads from M".
>
> I'm hand-waving on some details, especially since write-throughs are
> supposed to be transparent to the client (is that right?), and this
> plan would mean they aren't quite transparent. One possibility:
>
> We could have a new server error code, used only by read-only mirrors,
> that says: "You tried to update to a revision higher than my youngest
> revision; maybe you should try reading from this other repository over
> here (the master)." The client would redirect its communications just
> for this RA session, and not remember the master permanently, since
> presumably the slave will update soon anyway.
>
> That would, in effect, implement the above-proposed rule.
It seems like if the server can seamlessly re-direct a commit to the
master server, it could potentially do the same for an update, when it
detected the need.
That being said, doing so might cover up that there is a problem with
the slave server that no one realizes exists. It seems like if the
client got an error, but was no worse for it once the slave was synced
would be the ideal.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before a release can happen
Posted by Karl Fogel <kf...@red-bean.com>.
Peter McNab <mc...@optushome.com.au> writes:
> So long as R+1 is guaranteed to be the result of this clients commit
> and not from another clients simultaneous action.
That shouldn't be an issue. I should have said "R+X", where X is the
difference between R and whatever revision the client created in the
master. Usually, X will be 1, but it might be higher. It doesn't
matter, since whatever value X has, the client will know it.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by Peter McNab <mc...@optushome.com.au>.
Karl Fogel wrote:
> "Mark Phippard" <ma...@gmail.com> writes:
>
>>> Generally it works well, however when the commit is "large" (I'm not sure,
>>> several megabytes or more or at least the commit time of a few minutes
>>> (???)) then we usually have problems where the slave has not synced up
>>> with
>>> the master yet and the client that committed it to the slave gets confused
>>> and starts giving errors. It has been possible but problematic to get
>>> things straightened out, and sometimes I've had a svnsync not go through
>>> and had to manually kick it and/or delete the svn:lock (I think that is
>>> the property) to get things going again.
>>>
>>> Is there a way to do a synchronous sync with the slave or some code magic
>>> behind the scenes that could be added in the client or slave server or
>>> master server to tell the client to wait a little longer to be synced
>>> up with the slave server correctly?
>>>
>>> I think what is happening is that the slave says "I've committed a certain
>>> version" and then at the end of the commit it checks that the slave server
>>> has that version which hasn't been synced yet and it gets confused.
>>>
>>> Thankfully this has not happened a whole lot but it has happened on
>>> multiple occasions. I'm telling my users that when they have a "large"
>>> commit (size or time is hard to determine) then to switch to the master to
>>> commit it and then switch back to the slave again.
>>>
>>> Any help, ideas, pointers, would be appreciated.
>>>
>> Seems like a problem that should be identified and fixed before we
>> release. Now that we have svnsync it would be super-cool if it could
>> be formally integrated into this so that you did not need a hook at
>> all.
>>
>
> A client-side solution might work. Take a client that's reading from
> a slave repository S, but whose commits go to a master M.
>
> 1. S is at revision R, M is at revision R
>
> 2. Now the client commits. Afterwards, M is at revision R+1, but S
> is still at revision R. (Assume the commit is large, so S can't
> sync up immediately.)
>
> 3. If the client knows that its commit went through to M, it can
> obey a rule that says: "Until S is at R+1 or higher, do all
> reads from M".
>
>
So long as R+1 is guaranteed to be the result of this clients commit and
not from another clients simultaneous action.
Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before a release can happen
Posted by Karl Fogel <kf...@red-bean.com>.
"Mark Phippard" <ma...@gmail.com> writes:
>> Generally it works well, however when the commit is "large" (I'm not sure,
>> several megabytes or more or at least the commit time of a few minutes
>> (???)) then we usually have problems where the slave has not synced up
>> with
>> the master yet and the client that committed it to the slave gets confused
>> and starts giving errors. It has been possible but problematic to get
>> things straightened out, and sometimes I've had a svnsync not go through
>> and had to manually kick it and/or delete the svn:lock (I think that is
>> the property) to get things going again.
>>
>> Is there a way to do a synchronous sync with the slave or some code magic
>> behind the scenes that could be added in the client or slave server or
>> master server to tell the client to wait a little longer to be synced
>> up with the slave server correctly?
>>
>> I think what is happening is that the slave says "I've committed a certain
>> version" and then at the end of the commit it checks that the slave server
>> has that version which hasn't been synced yet and it gets confused.
>>
>> Thankfully this has not happened a whole lot but it has happened on
>> multiple occasions. I'm telling my users that when they have a "large"
>> commit (size or time is hard to determine) then to switch to the master to
>> commit it and then switch back to the slave again.
>>
>> Any help, ideas, pointers, would be appreciated.
>
> Seems like a problem that should be identified and fixed before we
> release. Now that we have svnsync it would be super-cool if it could
> be formally integrated into this so that you did not need a hook at
> all.
A client-side solution might work. Take a client that's reading from
a slave repository S, but whose commits go to a master M.
1. S is at revision R, M is at revision R
2. Now the client commits. Afterwards, M is at revision R+1, but S
is still at revision R. (Assume the commit is large, so S can't
sync up immediately.)
3. If the client knows that its commit went through to M, it can
obey a rule that says: "Until S is at R+1 or higher, do all
reads from M".
I'm hand-waving on some details, especially since write-throughs are
supposed to be transparent to the client (is that right?), and this
plan would mean they aren't quite transparent. One possibility:
We could have a new server error code, used only by read-only mirrors,
that says: "You tried to update to a revision higher than my youngest
revision; maybe you should try reading from this other repository over
here (the master)." The client would redirect its communications just
for this RA session, and not remember the master permanently, since
presumably the slave will update soon anyway.
That would, in effect, implement the above-proposed rule.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before a release can happen
Posted by Mark Phippard <ma...@gmail.com>.
On 5/20/07, David Summers <da...@summersoft.fay.ar.us> wrote:
>
> On Sun, 20 May 2007, Mark Phippard wrote:
>
> > DAV write-through proxy
> > This has the potential to be a killer feature, but it is another one
> > that is not heavily documented. There is a notes file, but it looks
> > like it pre-dates the implementation in some of the details. This one
> > looks fairly easy to turn on but could benefit from some more
> > explanation about what it wants to accomplish with some real-world
> > implementation scenarios. For example, it seems fairly important that
> > the read-only "slaves" are updated after a commit. Presumably svnsync
> > would be a good candidate for doing this. An example hook-based
> > implementation would be useful. Also, some discussion on the
> > ramifications would be useful. What if I do a commit, followed by an
> > update and my mirror has not been updated yet? Likewise, what if
> > another user does a commit, and then I do a commit by mirror has not
> > been updated yet. Will that effect me?
> >
>
> I have been using Subversion 1.4.2 on a Win2K3R2 master server with
> Subversion 1.5 r???? (I forget, updated last sometime in January)
> pre-release on a Linux Redhat 4 server since mid-December.
I was hoping you would reply since AFAIK, you are one of the few/only
that have been using it in advance of the release.
> I have a post-commit script that does svnsync from the master to the
> server and do web-dav proyxing (is that the correct term?) on the slave
> server so we can commit to the slave and it re-directs the commit to the
> master.
How does the script work? Could you make a sample version of it and
add it to the file in notes?
> Generally it works well, however when the commit is "large" (I'm not sure,
> several megabytes or more or at least the commit time of a few minutes
> (???)) then we usually have problems where the slave has not synced up
> with
> the master yet and the client that committed it to the slave gets confused
> and starts giving errors. It has been possible but problematic to get
> things straightened out, and sometimes I've had a svnsync not go through
> and had to manually kick it and/or delete the svn:lock (I think that is
> the property) to get things going again.
>
> Is there a way to do a synchronous sync with the slave or some code magic
> behind the scenes that could be added in the client or slave server or
> master server to tell the client to wait a little longer to be synced
> up with the slave server correctly?
>
> I think what is happening is that the slave says "I've committed a certain
> version" and then at the end of the commit it checks that the slave server
> has that version which hasn't been synced yet and it gets confused.
>
> Thankfully this has not happened a whole lot but it has happened on
> multiple occasions. I'm telling my users that when they have a "large"
> commit (size or time is hard to determine) then to switch to the master to
> commit it and then switch back to the slave again.
>
> Any help, ideas, pointers, would be appreciated.
Seems like a problem that should be identified and fixed before we
release. Now that we have svnsync it would be super-cool if it could
be formally integrated into this so that you did not need a hook at
all.
> It is a VERY useful feature and it generally works well.
I agree that it could be a very important feature. I'd like to see
someone pick it up and try to polish it up some more so that we could
really promote people to use it when the need exists.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by David Summers <da...@summersoft.fay.ar.us>.
On Sun, 20 May 2007, Mark Phippard wrote:
> DAV write-through proxy
> This has the potential to be a killer feature, but it is another one
> that is not heavily documented. There is a notes file, but it looks
> like it pre-dates the implementation in some of the details. This one
> looks fairly easy to turn on but could benefit from some more
> explanation about what it wants to accomplish with some real-world
> implementation scenarios. For example, it seems fairly important that
> the read-only "slaves" are updated after a commit. Presumably svnsync
> would be a good candidate for doing this. An example hook-based
> implementation would be useful. Also, some discussion on the
> ramifications would be useful. What if I do a commit, followed by an
> update and my mirror has not been updated yet? Likewise, what if
> another user does a commit, and then I do a commit by mirror has not
> been updated yet. Will that effect me?
>
I have been using Subversion 1.4.2 on a Win2K3R2 master server with
Subversion 1.5 r???? (I forget, updated last sometime in January)
pre-release on a Linux Redhat 4 server since mid-December.
I have a post-commit script that does svnsync from the master to the
server and do web-dav proyxing (is that the correct term?) on the slave
server so we can commit to the slave and it re-directs the commit to the
master.
Generally it works well, however when the commit is "large" (I'm not sure,
several megabytes or more or at least the commit time of a few minutes
(???)) then we usually have problems where the slave has not synced up
with
the master yet and the client that committed it to the slave gets confused
and starts giving errors. It has been possible but problematic to get
things straightened out, and sometimes I've had a svnsync not go through
and had to manually kick it and/or delete the svn:lock (I think that is
the property) to get things going again.
Is there a way to do a synchronous sync with the slave or some code magic
behind the scenes that could be added in the client or slave server or
master server to tell the client to wait a little longer to be synced
up with the slave server correctly?
I think what is happening is that the slave says "I've committed a certain
version" and then at the end of the commit it checks that the slave server
has that version which hasn't been synced yet and it gets confused.
Thankfully this has not happened a whole lot but it has happened on
multiple occasions. I'm telling my users that when they have a "large"
commit (size or time is hard to determine) then to switch to the master to
commit it and then switch back to the slave again.
Any help, ideas, pointers, would be appreciated.
It is a VERY useful feature and it generally works well.
Thanks!
--
David Wayne Summers "Linux: Because reboots are for hardware upgrades!"
david@summersoft.fay.ar.us PGP Key: http://summersoft.fay.ar.us/~david/pgp.txt
PGP Key fingerprint = 0B44 B118 85CC F4EC 7021 1ED4 1516 5B78 E320 2001
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by Vlad Georgescu <vg...@gmail.com>.
Malcolm Rowe wrote:
> On Fri, May 25, 2007 at 06:50:00PM +0300, Vlad Georgescu wrote:
>> The client supports the CRAM-MD5 and ANONYMOUS mechanisms (these are
>> built into Subversion, so you don't need the corresponding SASL
>> plugins), plus any other mechanisms that SASL was built/installed with.
>>
>
> I think it's worth calling that out in the notes: you can't disable
> ANONYMOUS and CRAM-MD5 by not having the SASL plugins (which is good,
> because a fair number of distros seem to ship with SASL+no plugins by
> default).
Done in r25176, thanks.
--
Vlad
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address
before a release can happen
Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Fri, May 25, 2007 at 06:50:00PM +0300, Vlad Georgescu wrote:
> The client supports the CRAM-MD5 and ANONYMOUS mechanisms (these are
> built into Subversion, so you don't need the corresponding SASL
> plugins), plus any other mechanisms that SASL was built/installed with.
>
I think it's worth calling that out in the notes: you can't disable
ANONYMOUS and CRAM-MD5 by not having the SASL plugins (which is good,
because a fair number of distros seem to ship with SASL+no plugins by
default).
Regards,
Malcolm
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by Vlad Georgescu <vg...@gmail.com>.
Mark Phippard wrote:
> On 5/25/07, Vlad Georgescu <vg...@gmail.com> wrote:
>> Mark Phippard wrote:
>>> [...]
>>> SASL
>>> Where is the notes file that explains the feature? What can you do
>>> with it? Does it, or does it not support TLS or SSL? Does it work on
>>> Windows? What is involved in bulding support for it into the client
>>> and server? What happens if client and server do not both have the
>>> support compiled in? How does one set it up etc. Given that we are
>>> reasonably close to a 1.5 release this information should all be
>>> readily available.
>> I added a file (notes/sasl.txt) in r25148 that I think addresses these
>> issues. Reviews are welcome.
>
> Thanks, it is a good document. I do have some questions/comments.
>
> 1) What controls which auth mechanisms the client supports? It must
> be something about how SASL that the client is using was compiled?
The client supports the CRAM-MD5 and ANONYMOUS mechanisms (these are
built into Subversion, so you don't need the corresponding SASL
plugins), plus any other mechanisms that SASL was built/installed with.
> 2) It seems like we really need to provide a pre-built library for
> Windows. There has been talk about doing this for all our
> dependencies which would be a good idea.
Yes, we certainly need a pre-built SASL library for Windows, with as
many plugins as possible. We should also add the SASL library and
plugins to the installer in order to be able to set SASL's registry keys
at installation time.
On Windows you should be able to do some cool things, like use the
GSSAPI mechanism to do single sign-on authentication against Active
Directory. This will only happen if we provide pre-built binaries.
> 3) As I think I understand it, the auth mechanism is not bound to the
> back-end store being used? So for example, you could use LDAP as the
> user store, and still support the CRAM-MD5 auth mechanism which would
> allow older clients to connect using their LDAP credentials? Server
> could also offer additional auth mechanisms for the newer clients?
I think SASL will simply "forward" authentication to the LDAP server, so
you should be able to use CRAM-MD5 with LDAP (LDAP also uses SASL), even
with older clients. Newer clients will be able to use the more advanced
mechanisms.
> 4) Encryption. Is there any other configuration involved, such as
> certificates? What exactly is encrypted? The svnserve traffic? Just
> the auth traffic? Just the post-auth traffic?
Just the post-auth traffic. No certificates are required, the
encryption key is derived from the password.
--
Vlad
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before a release can happen
Posted by Mark Phippard <ma...@gmail.com>.
On 5/25/07, Vlad Georgescu <vg...@gmail.com> wrote:
> Mark Phippard wrote:
> > [...]
> > SASL
> > Where is the notes file that explains the feature? What can you do
> > with it? Does it, or does it not support TLS or SSL? Does it work on
> > Windows? What is involved in bulding support for it into the client
> > and server? What happens if client and server do not both have the
> > support compiled in? How does one set it up etc. Given that we are
> > reasonably close to a 1.5 release this information should all be
> > readily available.
>
> I added a file (notes/sasl.txt) in r25148 that I think addresses these
> issues. Reviews are welcome.
Thanks, it is a good document. I do have some questions/comments.
1) What controls which auth mechanisms the client supports? It must
be something about how SASL that the client is using was compiled?
2) It seems like we really need to provide a pre-built library for
Windows. There has been talk about doing this for all our
dependencies which would be a good idea.
3) As I think I understand it, the auth mechanism is not bound to the
back-end store being used? So for example, you could use LDAP as the
user store, and still support the CRAM-MD5 auth mechanism which would
allow older clients to connect using their LDAP credentials? Server
could also offer additional auth mechanisms for the newer clients?
4) Encryption. Is there any other configuration involved, such as
certificates? What exactly is encrypted? The svnserve traffic? Just
the auth traffic? Just the post-auth traffic?
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: New (1.5) feature questions -- things we need to address before
a release can happen
Posted by Vlad Georgescu <vg...@gmail.com>.
Mark Phippard wrote:
> [...]
> SASL
> Where is the notes file that explains the feature? What can you do
> with it? Does it, or does it not support TLS or SSL? Does it work on
> Windows? What is involved in bulding support for it into the client
> and server? What happens if client and server do not both have the
> support compiled in? How does one set it up etc. Given that we are
> reasonably close to a 1.5 release this information should all be
> readily available.
I added a file (notes/sasl.txt) in r25148 that I think addresses these
issues. Reviews are welcome.
--
Vlad
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org