You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Lutz Frommberger <lu...@informatik.uni-bremen.de> on 2005/06/06 10:41:57 UTC
Post commit hook script: strange behavior
Hi all,
I am wondering if there is anything special about the behavior of the
hook scripts. I once noticed that scripts invoked by "post-commit" run
about twice as long as if they are invoked by cron. And now I have
another, fatal problem:
The script invoked by the hook script hangs, and the commit does not
finish at remote side. The line the script stops at is:
cat keywords.txt | xargs -P 1 -n 2 --replace=KEY /usr/bin/bib2bib -c \
'keywords: "KEY"' ./r3-literature.bib -ob KEY.bib
and the process list shows: (sorry for the formatting)
www-data 23324 0.0 0.4 2472 1128 ? S 12:28 0:00 /bin/sh /srv/svn/r3/r3-literature/hooks/post-commit /srv/svn/r3/r3-literature 129
www-data 23325 0.0 0.4 2468 1148 ? S 12:28 0:00 /bin/sh /home/r3/bin/makebibpages
www-data 23332 0.0 0.1 1492 416 ? S 12:28 0:00 xargs -P 1 -n 2 --replace=KEY /usr/bin/bib2bib -c keywords: "KEY" ./r3-literature.bib -ob KEY.bib
www-data 23357 0.0 0.4 2680 1076 ? S 12:28 0:00 /usr/bin/ocamlrun /usr/bin/bib2bib -c keywords: "robotics" ./r3-literature.bib -ob robotics.bib
If calling post-commit from a shell (as user www-data), the script works
without a problem. Also the script runs without a problem when invoked
by cron (as the same user). Hardware problems are very unlikely, the
server moved to new hardware recently without changing this behavior.
So where is the speciality about the post commit thing? Do I understand
something wrong?
System specs:
Linux version 2.6.8-1-386 (joshk@trollwife) (gcc version 3.3.5 (Debian
1:3.3.5-2)) #1 Thu Nov 25 04:24:08 UTC 2004
svn, version 1.1.4 (r13838)
compiled May 13 2005, 06:29:47
Apache 2
gruß,
--
Lutz Frommberger
SFB/TR 8 Spatial Cognition - Project R3-[Q-Shape]
Cognitive Systems Research Group, Universität Bremen
http://www.cosy.informatik.uni-bremen.de/staff/lutz/
Re: Post commit hook script: strange behavior
Posted by Lutz Frommberger <lu...@informatik.uni-bremen.de>.
Am Montag, den 06.06.2005, 16:57 +0200 schrieb Hannes Erven:
> I described a similar symptom at
> http://svn.haxx.se/users/archive-2005-05/1556.shtml , but despite a "me
> too" there was no reply on the list.
This sounds very similar, yes.
> How do you recover from that situation? Can you use the repository "as
> usual" while the hook script is hanging?
I never really tried that. But the hook script itself performs a
successful svn update, and the web frontend works. I end up in killing
all related processes on the server.
> I have a feeling that we might share the same problem source. ;-)
The problem seems to be: hook scripts sometimes behave strangely. No
idea how to come to a solution...
regards,
--
Lutz Frommberger
SFB/TR 8 Spatial Cognition - Project R3-[Q-Shape]
Cognitive Systems Research Group, Universität Bremen
http://www.cosy.informatik.uni-bremen.de/staff/lutz/
Re: Post commit hook script: strange behavior
Posted by Hannes Erven <ha...@erven.at>.
Hi Lutz,
> The script invoked by the hook script hangs, and the commit does not
> finish at remote side. The line the script stops at is:
I described a similar symptom at
http://svn.haxx.se/users/archive-2005-05/1556.shtml , but despite a "me
too" there was no reply on the list.
How do you recover from that situation? Can you use the repository "as
usual" while the hook script is hanging?
I have a feeling that we might share the same problem source. ;-)
Liebe Grüße,
-hannes
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Post commit hook script: strange behavior
Posted by Lutz Frommberger <lu...@informatik.uni-bremen.de>.
Am Montag, den 06.06.2005, 13:16 +0200 schrieb Nicolas Goutte:
> The FAQ tells that such behaviour is normally due to $PATH missing. Have you
> checked that it is not the problem.
Yes, of course. The script sets $PATH according to the needs.
> One special thing is that the environment is empty (for security reasons).
> (However for crontab $PATH is mostly defined.)
The problem is even stranger: The script _stops_ somewhere in the middle
while executing
cat keywords.txt|xargs -P 1 -n 2 --replace=KEY /usr/bin/bib2bib -c \
'keywords: "KEY"' ./r3-literature.bib -ob KEY.bib
The first 9 or 10 lines of keywords.txt are processed, and then it
suddenly stops.
regards,
--
Lutz Frommberger
SFB/TR 8 Spatial Cognition - Project R3-[Q-Shape]
Cognitive Systems Research Group, Universität Bremen
http://www.cosy.informatik.uni-bremen.de/staff/lutz/
Re: Post commit hook script: strange behavior
Posted by Marc Haisenko <ha...@webport.de>.
On Monday 06 June 2005 15:40, Lutz Frommberger wrote:
> But the ulimit hint perhaps explains the behavior that scripts doing
> much work invoked from post-commit run longer: the process limit is 2047
> then, and unlimited otherwise. Could be an explanation (but I still
> don't know why this occurs).
The process limit is the maximum number of processes allowed to run
simultanious ! This won't affect your execution speed and should not be set
to "unlimited" because simple fork bombs could bring your system down to a
halt (like this one: ":() { :|:& };:", try setting "ulimit -u 200" and run
this script as a user and then do "ulimit -u unlimited" and run it again...
but save your work first ;-)
C'ya,
Marc
--
Marc Haisenko
Systemspezialist
Webport IT-Services GmbH
mailto: haisenko@webport.de
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Post commit hook script: strange behavior
Posted by Nicolas Goutte <ni...@snafu.de>.
On Monday 06 June 2005 14:42, Lutz Frommberger wrote:
> Am Montag, den 06.06.2005, 14:30 +0200 schrieb Nicolas Goutte:
> > Just an idea: have you different ulimit values for the user running the
> > script from Subversion and that such a limit would be triggered?
> > (assuming Subversion is running as another user).
>
> The user is always "www-data", no matter where I call the script. But
> anyway, this could be a good idea, because
>
> $ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> file size (blocks, -f) unlimited
> max locked memory (kbytes, -l) unlimited
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) unlimited
> virtual memory (kbytes, -v) unlimited
>
> tells me a pipe size of 8 - and it works for files with no more than 8
> entries, and this is a piped call. Can't be chance...
Yes but 8 * 512 should be more than just 10 keywords or are your keywords so
large. (Also if the limit is reached, it porbably just means that cat is made
to sleep for a while, until th epipe has emptied again.
>
> My problem is that I never heard of the command "ulimit" before - what
> can I do to work around this problem? How can I (for testing) increase
> the pipe size? Even root seems to have a pipe size limit of 8.
As far as I understand ulimit -p cannot be set.
>
> Many thanks up to now.
>
> regards,
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Post commit hook script: strange behavior
Posted by Nicolas Goutte <ni...@snafu.de>.
On Monday 06 June 2005 16:23, Lutz Frommberger wrote:
> Am Montag, den 06.06.2005, 15:40 +0200 schrieb Lutz Frommberger:
> > I found a solution, which does not make me happy: the processes
> > (obviuosly) stopped because of whitespaces in the keyword list. The -0
> > option to xargs healed this.
>
> Sorry answering to myself: it did _not_ heal it. -0 just put everything
> into one line, resulting in a pipe of one single element. The script
> terminated, but did nothing.
Assuming that you entries are one by line, you could try:
xargs -eof="\n"
Have a nice day!
>
> Splitting the keyword list into separate files helped (smaller pipes),
> but then the problem occurs in the next line. So, nothing solved, still
> clueless. :-(
>
> (Increasing umlits for processes and stack size did not help either.)
>
> gruß,
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Post commit hook script: strange behavior
Posted by Lutz Frommberger <lu...@informatik.uni-bremen.de>.
Am Montag, den 06.06.2005, 15:40 +0200 schrieb Lutz Frommberger:
> I found a solution, which does not make me happy: the processes
> (obviuosly) stopped because of whitespaces in the keyword list. The -0
> option to xargs healed this.
Sorry answering to myself: it did _not_ heal it. -0 just put everything
into one line, resulting in a pipe of one single element. The script
terminated, but did nothing.
Splitting the keyword list into separate files helped (smaller pipes),
but then the problem occurs in the next line. So, nothing solved, still
clueless. :-(
(Increasing umlits for processes and stack size did not help either.)
gruß,
--
Lutz Frommberger
SFB/TR 8 Spatial Cognition - Project R3-[Q-Shape]
Cognitive Systems Research Group, Universität Bremen
http://www.cosy.informatik.uni-bremen.de/staff/lutz/
Re: Post commit hook script: strange behavior
Posted by Lutz Frommberger <lu...@informatik.uni-bremen.de>.
Am Montag, den 06.06.2005, 14:30 +0200 schrieb Nicolas Goutte:
> I mean especially the -t limit which limits the time used on the CPU. (Often
> for servers there are time limits, so that a process cannot bring down a
> server by an endless loop.)
I found a solution, which does not make me happy: the processes
(obviuosly) stopped because of whitespaces in the keyword list. The -0
option to xargs healed this.
Anyway, I am still wondering why this is a problem in a hook script, but
not from cron...
But the ulimit hint perhaps explains the behavior that scripts doing
much work invoked from post-commit run longer: the process limit is 2047
then, and unlimited otherwise. Could be an explanation (but I still
don't know why this occurs).
Thanks again for all hints.
regards,
--
Lutz Frommberger
SFB/TR 8 Spatial Cognition - Project R3-[Q-Shape]
Cognitive Systems Research Group, Universität Bremen
http://www.cosy.informatik.uni-bremen.de/staff/lutz/
Re: Post commit hook script: strange behavior
Posted by Lutz Frommberger <lu...@informatik.uni-bremen.de>.
Am Montag, den 06.06.2005, 14:30 +0200 schrieb Nicolas Goutte:
> Just an idea: have you different ulimit values for the user running the script
> from Subversion and that such a limit would be triggered? (assuming
> Subversion is running as another user).
The user is always "www-data", no matter where I call the script. But
anyway, this could be a good idea, because
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
tells me a pipe size of 8 - and it works for files with no more than 8
entries, and this is a piped call. Can't be chance...
My problem is that I never heard of the command "ulimit" before - what
can I do to work around this problem? How can I (for testing) increase
the pipe size? Even root seems to have a pipe size limit of 8.
Many thanks up to now.
regards,
--
Lutz Frommberger
SFB/TR 8 Spatial Cognition - Project R3-[Q-Shape]
Cognitive Systems Research Group, Universität Bremen
http://www.cosy.informatik.uni-bremen.de/staff/lutz/
Re: Post commit hook script: strange behavior
Posted by Nicolas Goutte <ni...@snafu.de>.
On Monday 06 June 2005 13:52, Lutz Frommberger wrote:
> Am Montag, den 06.06.2005, 13:16 +0200 schrieb Nicolas Goutte:
> > The FAQ tells that such behaviour is normally due to $PATH missing. Have
> > you checked that it is not the problem.
>
> Yes, of course. The script sets $PATH according to the needs.
>
> > One special thing is that the environment is empty (for security
> > reasons). (However for crontab $PATH is mostly defined.)
>
> The problem is even stranger: The script _stops_ somewhere in the middle
> while executing
Just an idea: have you different ulimit values for the user running the script
from Subversion and that such a limit would be triggered? (assuming
Subversion is running as another user).
I mean especially the -t limit which limits the time used on the CPU. (Often
for servers there are time limits, so that a process cannot bring down a
server by an endless loop.)
Have a nice day!
>
> cat keywords.txt|xargs -P 1 -n 2 --replace=KEY /usr/bin/bib2bib -c \
> 'keywords: "KEY"' ./r3-literature.bib -ob KEY.bib
>
> The first 9 or 10 lines of keywords.txt are processed, and then it
> suddenly stops.
>
> regards,
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Post commit hook script: strange behavior
Posted by Nicolas Goutte <ni...@snafu.de>.
On Monday 06 June 2005 12:41, Lutz Frommberger wrote:
(...)
>
> If calling post-commit from a shell (as user www-data), the script works
> without a problem. Also the script runs without a problem when invoked
> by cron (as the same user). Hardware problems are very unlikely, the
> server moved to new hardware recently without changing this behavior.
The FAQ tells that such behaviour is normally due to $PATH missing. Have you
checked that it is not the problem.
>
> So where is the speciality about the post commit thing? Do I understand
> something wrong?
One special thing is that the environment is empty (for security reasons).
(However for crontab $PATH is mostly defined.)
>
(...)
Have a nice day!
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Post commit hook script: strange behavior
Posted by Lutz Frommberger <lu...@informatik.uni-bremen.de>.
Am Montag, den 06.06.2005, 14:31 +0200 schrieb Ryan Schmidt:
> Have you tried /bin/cat and /usr/bin/xargs (or whatever their paths are
> on your system) instead of just cat and xargs?
Not yet - I don't think that's the reason. The script works fine when
there are about 8 entries in keywords.txt, even from post-commit. It
just fails if there are more.
regards,
--
Lutz Frommberger
SFB/TR 8 Spatial Cognition - Project R3-[Q-Shape]
Cognitive Systems Research Group, Universität Bremen
http://www.cosy.informatik.uni-bremen.de/staff/lutz/
Re: Post commit hook script: strange behavior
Posted by Ryan Schmidt <su...@ryandesign.com>.
On 06.06.2005, at 12:41, Lutz Frommberger wrote:
> The script invoked by the hook script hangs, and the commit does not
> finish at remote side. The line the script stops at is:
>
> cat keywords.txt | xargs -P 1 -n 2 --replace=KEY /usr/bin/bib2bib -c \
> 'keywords: "KEY"' ./r3-literature.bib -ob KEY.bib
[snip]
> If calling post-commit from a shell (as user www-data), the script
> works
> without a problem. Also the script runs without a problem when invoked
> by cron (as the same user). Hardware problems are very unlikely, the
> server moved to new hardware recently without changing this behavior.
Have you tried /bin/cat and /usr/bin/xargs (or whatever their paths are
on your system) instead of just cat and xargs?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org