You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Ray Rashif <sc...@archaudio.org> on 2011/08/02 22:03:35 UTC
Subversion transactions within post-commit
Hi list
I've been stumped by this issue for months. I had never enough time to
troubleshoot it, so I'm finally desperate after having spent an entire
day trying to figure things out.
We have a _local_ checkout of the repository on our server, and our
post-commit hook needs to run svn-add, svn-rm and svn-ci commands in
that checkout. All of these commands run fine. For eg. (within the
post-commit script):
echo "Updating..."
svn up
would yield:
Updating...
Updated to revision $whatever.
And:
echo "Adding..."
svn add
would output:
Adding...
A $whatever
Now comes the interesting part. As soon as I add a commit transaction,
say, AFTER those commands, ALL of them FAIL at the SAME TIME:
echo "Updating.."
svn up
echo "Adding..."
svn add $whatever
echo "Committing..."
svn ci -m "$whatever"
gives us:
Updating...
svn: Working copy '.' locked
svn: run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)
Adding...
svn: Working copy '.' locked
svn: run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)
Committing...
svn: Working copy '.' locked
svn: run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)
Notes:
* It is always reproducible. I just remove the commit command from the
script and everything works as expected.
* The owner of the checked out repo is a normal user of the web
account, but post-commit is run by 'apache'.
* We have the repo chown'ed to $user:$user and perms set to 775.
* The user 'apache' is also part of the group $user, but the user
$user is _not_ part of the group 'apache' (not possible for security
reasons on the server).
* As such, both the currently logged in user and 'apache' can at least
do update transactions without any errors.
Another very important note:
* Since post-commit runs as 'apache', some Subversion files (not sure
which) are created with perms 444.
Would appreciate some help in this matter. Thanks!
--
GPG/PGP ID: 19BAA7AD
Re: Subversion transactions within post-commit
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Tue, Aug 2, 2011 at 6:49 PM, Ray Rashif <sc...@archaudio.org> wrote:
> On 3 August 2011 05:47, Nico Kadel-Garcia <nk...@gmail.com> wrote:
>> That's what "svn checkout" for initial setup, and "svn update" or "svn
>> revert" of the local copy are for.
>
> We already have that working, I just outlined the current setup:
>
> http://files.foobar.org/ is a direct file mirror (less revision
> control; plus additional files generated post-commit dependent on the
> latest version of each revision controlled file) of
> http://svn.foobar.org/trunk/somewhere.
>
> Developers push stuff to somewhere (which is private in Subversion).
> This private area then gets served as HTTP/FTP, edited appropriately
> for public use (and also for programs as some of the files generated
> after a commit are databases).
>
>> Then you do your updates in a working copy as part of a working copy,
>> and that copy has a deployment script or "Makefile" that does "rsync"
>> or other mirroring tools to publish those changes to your exposed
>> website. I like "Makefile" bassed approaches, myself, so I can do
>> "make", "make test", and "make install".
>
> Ahh yes. This was one of our earlier plans, but I scraped that because
> it would duplicate the contents (of the SVN repo) on the server. But
> let me get this right:
>
> svn commit (anywhere) -> post-commit -> svn up (local checkout), rsync
> -> local mirror -> edits ?
Put it in a checklist
1: svn commit (from any authorized user)
2: post-commit publishes updates
I) Create designated working copy, if needed
II) "svn revert" in working copy, to avlid conflicting local
changes overlapping push process
iii) "svn status" in working copy, to report failures or conflicting debris
iv) "svn update" to get relevant updates
v) Edit or generate non-committed local files as needed
vi) Push working copy contents to website with some efficient tool
that does not overwrite correctly published content, such as "rsync"
vii) Leave working copy around for future updates as appropriate
Re: Subversion transactions within post-commit
Posted by Ray Rashif <sc...@archaudio.org>.
On 3 August 2011 05:47, Nico Kadel-Garcia <nk...@gmail.com> wrote:
> That's what "svn checkout" for initial setup, and "svn update" or "svn
> revert" of the local copy are for.
We already have that working, I just outlined the current setup:
http://files.foobar.org/ is a direct file mirror (less revision
control; plus additional files generated post-commit dependent on the
latest version of each revision controlled file) of
http://svn.foobar.org/trunk/somewhere.
Developers push stuff to somewhere (which is private in Subversion).
This private area then gets served as HTTP/FTP, edited appropriately
for public use (and also for programs as some of the files generated
after a commit are databases).
> Then you do your updates in a working copy as part of a working copy,
> and that copy has a deployment script or "Makefile" that does "rsync"
> or other mirroring tools to publish those changes to your exposed
> website. I like "Makefile" bassed approaches, myself, so I can do
> "make", "make test", and "make install".
Ahh yes. This was one of our earlier plans, but I scraped that because
it would duplicate the contents (of the SVN repo) on the server. But
let me get this right:
svn commit (anywhere) -> post-commit -> svn up (local checkout), rsync
-> local mirror -> edits ?
--
GPG/PGP ID: 19BAA7AD
Re: Subversion transactions within post-commit
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Tue, Aug 2, 2011 at 5:00 PM, Ray Rashif <sc...@archaudio.org> wrote:
> On 3 August 2011 04:37, David Weintraub <qa...@gmail.com> wrote:
>> NO! NO! NO! AUUUGGGHHH!
>>
>> Never update a repository with a hook.! That's why you're given the
>> relatively safe "svnlook" command. It allows you to look, but not
>> touch a repository.
>> ...
>> SNIP
>> ...
>
> Arrrgghhh damn it! Exactly what I feared. Especially the recursion. So
> much so that now I think I was in denial all the while.
>
> OK, looks like I have to explain a bit further to clear this up and
> hopefully get some solutions.
>
> We use the repository to directly serve some files (so Subversion
> updates are immediately reflected). Something like
> http://subversion.apache.org/faq.html#website-auto-update
>
> So $files.subdomain.org/public_html is actually a local checkout of
> $svn.subdomain.org/public_html/trunk/$somedir, and whenever someone
> commits something to the SVN repo, the post-commit hook updates the
> local checkout. So anyone browsing the file server will get the most
> recent files.
That's what "svn checkout" for initial setup, and "svn update" or "svn
revert" of the local copy are for.
> However, we need to be a bit more personal than that. We need to do
> some checks on those files and remove ones that should not be there.
> We need to do this immediately after the commit, so cron is not an
> option. How should we go about doing this?
Then you do your updates in a working copy as part of a working copy,
and that copy has a deployment script or "Makefile" that does "rsync"
or other mirroring tools to publish those changes to your exposed
website. I like "Makefile" bassed approaches, myself, so I can do
"make", "make test", and "make install".
>
> --
> GPG/PGP ID: 19BAA7AD
>
Re: Subversion transactions within post-commit
Posted by David Weintraub <qa...@gmail.com>.
On Wed, Aug 3, 2011 at 2:31 PM, Ray Rashif <sc...@archaudio.org> wrote:
> On 4 August 2011 01:59, David Weintraub <qa...@gmail.com> wrote:
> Thanks guys. I'm just stupid. I've been going on and on about what we
> want to do, but I haven't even stated the obvious:
>
> * It's just about removing stale files from being tracked by
> Subversion, _without_ committer intervention.
>
> Basically, our developers (did|can do) this thing manually (remove an
> older version of a binary they are committing). I wanted to automate
> this. That's all!
Is this binary file a file that's built out of your source code? If it
is, it shouldn't be in your repository at all. Instead, it should be
built when the deployment is needed.
That's what Jenkins and Hudson can do for you. When the code is
checked in, they'll do the build, recreate the binary, and do
deployment automatically. Your developers can go to the Jenkins/Hudson
webpage and download the deployment including the built files. Since
Jenkins/Hudson can store multiple builds, there's no need for any
built objects to be stored in your source code.
Jenkins and Hudson are simple to setup, easy to use, yet very powerful
tools. Take a look at http://jenkins-ci.org.
(Jenkins and Hudson are a forks off the original Sun project called
Hudson. Oracle bought Sun, and the main developer of Hudson left and
took the code with him as well as most of the other developers. Oracle
owns the Hudson name, so the developers renamed the project Jenkins.
Now, both projects claim they're the original and the other is a mere
fork. As of now, most of the activity is on Jenkins which is what I
use.)
--
David Weintraub
qazwart@gmail.com
Re: Subversion transactions within post-commit
Posted by Ray Rashif <sc...@archaudio.org>.
On 4 August 2011 01:59, David Weintraub <qa...@gmail.com> wrote:
> ...
> SNIP
> ...
> I have what I call my Kitchen Sink pre-commit trigger that does all
> sorts of stuff. One of the tasks it can do is "ban" files based upon
> their name (either regular expression or Ant glob). If a user adds a
> file with a forbidden name, the commit is rejected and the user is
> informed of the offending file. The user has to remove the file from
> their commit and try again.
Thanks guys. I'm just stupid. I've been going on and on about what we
want to do, but I haven't even stated the obvious:
* It's just about removing stale files from being tracked by
Subversion, _without_ committer intervention.
Basically, our developers (did|can do) this thing manually (remove an
older version of a binary they are committing). I wanted to automate
this. That's all!
So, after the discussion here, it's evident I'm doing this all wrong.
I've now just reverted our stuff to the last best working state, and
I'll be working on something based on:
pre-commit (as much filtering as possible during this stage)
post-commit (update the checkout)
rsync (mirror the checkout for publishing; do not reflect stale files)
cron (to svn rm; svn ci the stale files)
--
GPG/PGP ID: 19BAA7AD
Re: Subversion transactions within post-commit
Posted by David Weintraub <qa...@gmail.com>.
On Tue, Aug 2, 2011 at 5:00 PM, Ray Rashif <sc...@archaudio.org> wrote:
> OK, looks like I have to explain a bit further to clear this up and
> hopefully get some solutions.
>
> We use the repository to directly serve some files (so Subversion
> updates are immediately reflected). Something like
> http://subversion.apache.org/faq.html#website-auto-update
>
> So $files.subdomain.org/public_html is actually a local checkout of
> $svn.subdomain.org/public_html/trunk/$somedir, and whenever someone
> commits something to the SVN repo, the post-commit hook updates the
> local checkout. So anyone browsing the file server will get the most
> recent files.
>
> However, we need to be a bit more personal than that. We need to do
> some checks on those files and remove ones that should not be there.
> We need to do this immediately after the commit, so cron is not an
> option. How should we go about doing this?
I would recommend using something like Jenkins/Hudson
(http://jenkins-ci.org). Jenkins will automatically be activated
whenever Jenkins detects a change in your Subversion repository for
that module. (Basically, whenever someone does a commit). Then, you
could have Jenkins do an export of the code, munge it, stop your
website, change the files, restart your website, and even run some
tests against the website. If there are any issues, Jenkins can mark
the commit as "bad" or "unstable" and then email the developers about
the issue.
Jenkins can run various scripting languages such as shell, MS-Batch,
Python, Perl, Ruby, use Ant, MSBuild, Nant, and I am sure a dozen
others I don't even know about.
I know this may not necessarily fix the problem, but it will alert you
of any issues. After a few days with Jenkins, developers will start to
be more careful about their commits since they don't like their name
associated with bad commits.
If you really need to keep certain files out of your Subversion
repository, you can use a pre-commit hook to make sure they are never
committed in the first place. If a commit contains one of the files
you want to keep out, the commit will fail and the user will have to
remove the files from their commit. It doesn't take long for the
developers to learn to be more careful.
I have what I call my Kitchen Sink pre-commit trigger that does all
sorts of stuff. One of the tasks it can do is "ban" files based upon
their name (either regular expression or Ant glob). If a user adds a
file with a forbidden name, the commit is rejected and the user is
informed of the offending file. The user has to remove the file from
their commit and try again.
--
David Weintraub
qazwart@gmail.com
Re: Subversion transactions within post-commit
Posted by Ray Rashif <sc...@archaudio.org>.
On 3 August 2011 04:37, David Weintraub <qa...@gmail.com> wrote:
> NO! NO! NO! AUUUGGGHHH!
>
> Never update a repository with a hook.! That's why you're given the
> relatively safe "svnlook" command. It allows you to look, but not
> touch a repository.
> ...
> SNIP
> ...
Arrrgghhh damn it! Exactly what I feared. Especially the recursion. So
much so that now I think I was in denial all the while.
OK, looks like I have to explain a bit further to clear this up and
hopefully get some solutions.
We use the repository to directly serve some files (so Subversion
updates are immediately reflected). Something like
http://subversion.apache.org/faq.html#website-auto-update
So $files.subdomain.org/public_html is actually a local checkout of
$svn.subdomain.org/public_html/trunk/$somedir, and whenever someone
commits something to the SVN repo, the post-commit hook updates the
local checkout. So anyone browsing the file server will get the most
recent files.
However, we need to be a bit more personal than that. We need to do
some checks on those files and remove ones that should not be there.
We need to do this immediately after the commit, so cron is not an
option. How should we go about doing this?
--
GPG/PGP ID: 19BAA7AD
Re: Subversion transactions within post-commit
Posted by David Weintraub <qa...@gmail.com>.
On Tue, Aug 2, 2011 at 4:03 PM, Ray Rashif <sc...@archaudio.org> wrote:
> We have a _local_ checkout of the repository on our server, and our
> post-commit hook needs to run svn-add, svn-rm and svn-ci commands in
> that checkout. All of these commands run fine. For eg. (within the
> post-commit script):
NO! NO! NO! AUUUGGGHHH!
Never update a repository with a hook.! That's why you're given the
relatively safe "svnlook" command. It allows you to look, but not
touch a repository.
Think of it this way, the user must wait until the post-commit hook
finishes. So, that means the user must wait until you've updated your
working directory, manipulated what you need, and then commit those
changes back.
Besides, there's the issue of recursion. Your hook script does a
commit which runs another hook script which runs another commit.
And, then there's the issue of a user making a set of changes, and
finding out that their working directory is out of date with changes
they didn't make.
Finally, there's the Finger 'o Blame which is dying to point to the
scapegoat who messed things up. You want the Finger 'o Blame to point
to the user who committed the transaction. Otherwise, you end up with
this:
Boss: Dammit, we have to delay the release and miss the release date.
Why was the code messed up?
Developer: It wasn't my fault. It worked on my machine. It was that
stupid post-commit hook that CM put on! It broke my code.
See, Finger o' Blame points right at you.
You didn't say what you're doing and why, so I can't say how you want
to do it. However, these usually fall into the category of "fixing"
the committed files. For example, you want to reformat the code to be
in the correct format. Instead, either put this as part of a
pre-commit trigger and simply fail the commit, or use a continuous
build system like Hudson or Jenkins to test the situation.
For example, users are suppose to run checkstyle on their code before
committing. You want to run checkstyle and fix the code. Instead, have
Jenkins or Hudson run Checkstyle, then mark the revision as bad and
email the user if checkstyle finds issues. Jenkins/Hudson will email
the user.
> Now comes the interesting part. As soon as I add a commit transaction,
> say, AFTER those commands, ALL of them FAIL at the SAME TIME:
>
> echo "Updating.."
> svn up
> echo "Adding..."
> svn add $whatever
> echo "Committing..."
> svn ci -m "$whatever"
>
> gives us:
>
> Updating...
> svn: Working copy '.' locked
> svn: run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)
> Adding...
> svn: Working copy '.' locked
> svn: run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)
> Committing...
> svn: Working copy '.' locked
> svn: run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)
Well, the repository is locked because another user is in the middle
of a commit transaction, and you're attempting to do another commit.
Although it is a post-commit hook, and the repository is already
modified, the commit is still going on.
See:
<http://svnbook.red-bean.com/en/1.5/svn.reposadmin.create.html#svn.reposadmin.create.hooks>
for more information.
--
David Weintraub
qazwart@gmail.com