You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by David James <ja...@gmail.com> on 2005/12/17 06:02:18 UTC
Fix make -j for SWIG bindings (was: swig: srcdir vs. blddir)
On 12/15/05, Philip Martin <ph...@codematters.co.uk> wrote:
> A more annoying problem is that the 'make -j4 swig-py' parallel build
> is broken, it sometimes fails with
>
> /usr/local/swig-1.3.24/bin/swig -I/home/pm/sw/subversion/obj/../svn/subversion/include -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig/include -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig/proxy -I/home/pm/sw/subversion/obj/subversion/bindings/swig/proxy -I/usr/local/apache2/include -I/usr/local/apache2/include -python -classic -w451 -w305 -o ../svn/subversion/bindings/swig/python/svn_client.c ../svn/subversion/bindings/swig/svn_client.i
> Traceback (most recent call last):
> File "build/generator/swig/external_runtime.py", line 107, in ?
> gen.write()
> File "build/generator/swig/external_runtime.py", line 99, in write
> self.write_long_long_fix()
> File "build/generator/swig/external_runtime.py", line 82, in write_long_long_fix
> self.checkout("python","python.swg")
> File "/home/pm/sw/subversion/svn/build/generator/swig/__init__.py", line 56, in checkout
> shutil.copy("%s/%s/%s" % (self.swig_libdir, dir, file), out)
> File "/usr/lib/python2.3/shutil.py", line 73, in copy
> copymode(src, dst)
> File "/usr/lib/python2.3/shutil.py", line 52, in copymode
> os.chmod(dst, mode)
> OSError: [Errno 2] No such file or directory: 'subversion/bindings/swig/proxy/python.swg'
Philip, does the attached patch solve the issue with make -j?
[[[
* build/generator/gen_make.py
Ensure that external_runtime.py is only called once. This fixes
"make -j", which crashes when multiple instances of
external_runtime.py try to create the same files.
Found by: philip
]]]
Cheers,
David
--
David James -- http://www.cs.toronto.edu/~james
Re: Fix make -j for SWIG bindings (was: swig: srcdir vs. blddir)
Posted by David James <ja...@gmail.com>.
On 12/17/05, Max Bowsher <ma...@ukf.net> wrote:
> David James wrote:
> > On 12/15/05, Philip Martin <ph...@codematters.co.uk> wrote:
> >
> >>A more annoying problem is that the 'make -j4 swig-py' parallel build
> >>is broken, it sometimes fails with
> >>
> >>/usr/local/swig-1.3.24/bin/swig -I/home/pm/sw/subversion/obj/../svn/subversion/include -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig/include -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig/proxy -I/home/pm/sw/subversion/obj/subversion/bindings/swig/proxy -I/usr/local/apache2/include -I/usr/local/apache2/include -python -classic -w451 -w305 -o ../svn/subversion/bindings/swig/python/svn_client.c ../svn/subversion/bindings/swig/svn_client.i
> >>Traceback (most recent call last):
> >> File "build/generator/swig/external_runtime.py", line 107, in ?
> >> gen.write()
> >> File "build/generator/swig/external_runtime.py", line 99, in write
> >> self.write_long_long_fix()
> >> File "build/generator/swig/external_runtime.py", line 82, in write_long_long_fix
> >> self.checkout("python","python.swg")
> >> File "/home/pm/sw/subversion/svn/build/generator/swig/__init__.py", line 56, in checkout
> >> shutil.copy("%s/%s/%s" % (self.swig_libdir, dir, file), out)
> >> File "/usr/lib/python2.3/shutil.py", line 73, in copy
> >> copymode(src, dst)
> >> File "/usr/lib/python2.3/shutil.py", line 52, in copymode
> >> os.chmod(dst, mode)
> >>OSError: [Errno 2] No such file or directory: 'subversion/bindings/swig/proxy/python.swg'
> >
> >
> > Philip, does the attached patch solve the issue with make -j?
> >
> > [[[
> >
> > * build/generator/gen_make.py
> > Ensure that external_runtime.py is only called once. This fixes
> > "make -j", which crashes when multiple instances of
> > external_runtime.py try to create the same files.
> >
> > Found by: philip
> >
> > ]]]
> >
> > Index: build/generator/gen_make.py
> > ===================================================================
> > --- build/generator/gen_make.py (revision 17814)
> > +++ build/generator/gen_make.py (working copy)
> > @@ -148,10 +148,16 @@
> > fname = '%s/swig_%s_external_runtime.swg' % (self.swig.proxy_dir, lang)
> > swig_runtime_fnames.append(fname)
> > if not self.release_mode:
> > + # external_runtime.py generates the SWIG runtimes for all languages
> > + # the first time it is called
> > + first_runtime = swig_runtime_fnames[0]
> > + other_runtimes = " ".join(swig_runtime_fnames[1:])
> > self.ofile.write(
> > - '%s:\n' % " ".join(swig_runtime_fnames) +
> > - '\tcd $(top_srcdir) && $(PYTHON)' +
> > + '%s: %s\n'
> > + '%s:\n'
> > + '\tcd $(top_srcdir) && $(PYTHON)'
> > ' build/generator/swig/external_runtime.py build.conf "$(SWIG)"\n\n'
> > + % (other_runtimes, first_runtime, first_runtime)
> > )
> >
> > self.ofile.write(
>
>
> Ouch! That's kind of inelegant. -0.9
>
> Exactly what is going on here?
>
> Surely it is a bug in 'make -j' if it executes the same make stages in
> multiple threads of control?
Here's a test case Makefile: {{{
all: a b c
a b c:
sleep 5 && touch a b c
}}}
If you run this Makefile without "-j", it executes the command once.
If you run it with "-j 4", it executes the touch command three times
in parallel. If you run multiple instances of our external_runtime.py
script in parallel, the three scripts will step on each other's toes,
trying to create and modify the same files at the same time.
Here's a new Makefile which avoids the problem: {{{
all: a b c
b c: a
a:
sleep 5 && touch a b c
}}}
My patch uses the same trick.
Cheers,
David
--
David James -- http://www.cs.toronto.edu/~james
Re: Fix make -j for SWIG bindings
Posted by Philip Martin <ph...@codematters.co.uk>.
David James <ja...@gmail.com> writes:
> Aha! This is exactly the problem. Using regular make, the
> "external_runtime.py" only gets invoked once because it generates all
> three files (plus some other files) the first time it is invoked.
> Using parallel make, it's possible that external_runtime.py could be
> run three times in parallel.
It's not even the fact that it runs in parallel that is the problem.
The script appears to delete the output files before recreating them,
so even sequential runs will delete files created by a previous run
thus explaining the 'No such file' error. Even without the delete,
the fact that the output files get rewritten means that while looking
at the results of a previous run a subsequent run could cause the file
to be incomplete.
> P.S. (It's also possible to fix this issue by updating
> external_runtime.py to generate each dependency, including the long
> long fix and the checked-out SWIG headers, separately. To implement
> this, we'd probably split up external_runtime.py into several scripts,
> including a 'swig_checkout.py'. Would anyone like to work on this?)
Yes, that would work. The current script is a very poor match for a
Makefile rule.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Fix make -j for SWIG bindings
Posted by David James <ja...@gmail.com>.
On 12/17/05, Philip Martin <ph...@codematters.co.uk> wrote:
> >> Philip, does the attached patch solve the issue with make -j?
>
> I haven't tried the patch.
>
> >> [[[
> >>
> >> * build/generator/gen_make.py
> >> Ensure that external_runtime.py is only called once. This fixes
> >> "make -j", which crashes when multiple instances of
> >> external_runtime.py try to create the same files.
> >>
> >> Found by: philip
> >>
> >> ]]]
> >>
> > Ouch! That's kind of inelegant. -0.9
> >
> > Exactly what is going on here?
>
> external_runtime.py gets invoked three times with exactly the same
> arguments because build-outputs.mk has three targets that invoke it
> (swig_python_external_runtime.swg, swig_perl_external_runtime.swg,
> swig_ruby_external_runtime.swg) and the swig-headers target depends on
> all three.
Aha! This is exactly the problem. Using regular make, the
"external_runtime.py" only gets invoked once because it generates all
three files (plus some other files) the first time it is invoked.
Using parallel make, it's possible that external_runtime.py could be
run three times in parallel.
Here's how I reproduced the bug. First, I edited build-outputs.mk to
sleep for 5 seconds before running external_runtime.py, so that I
could reliably reproduce the bug without worrying about timing issues.
Then I ran the following commands: {{{
make autogen-swig-pl
rm -f subversion/bindings/swig/proxy/*runtime.swg
make -j 3 autogen-swig-pl
}}}
Here's the output: {{{
james@syntax% make -j 3 autogen-swig /nobackup/clgrp/james/trunks
sleep 5 && cd . && /h/46/james/bin/python
build/generator/swig/external_runtime.py build.conf
"/h/46/james/bin/swig"
sleep 5 && cd . && /h/46/james/bin/python
build/generator/swig/external_runtime.py build.conf
"/h/46/james/bin/swig"
sleep 5 && cd . && /h/46/james/bin/python
build/generator/swig/external_runtime.py build.conf
"/h/46/james/bin/swig"
Traceback (most recent call last):
File "build/generator/swig/external_runtime.py", line 107, in ?
gen.write()
File "build/generator/swig/external_runtime.py", line 98, in write
self.write_external_runtime()
File "build/generator/swig/external_runtime.py", line 30, in
write_external_runtime
self.checkout(".","runtime.swg")
File "/nobackup/clgrp/james/trunks/build/generator/swig/__init__.py",
line 54, in checkout
os.remove(out)
OSError: [Errno 2] No such file or directory:
'subversion/bindings/swig/proxy/runtime.swg'
make: *** [subversion/bindings/swig/proxy/swig_ruby_external_runtime.swg]
Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
File "build/generator/swig/external_runtime.py", line 107, in ?
gen.write()
File "build/generator/swig/external_runtime.py", line 98, in write
self.write_external_runtime()
File "build/generator/swig/external_runtime.py", line 30, in
write_external_runtime
self.checkout(".","runtime.swg")
File "/nobackup/clgrp/james/trunks/build/generator/swig/__init__.py", line 56,
in checkout
shutil.copy("%s/%s/%s" % (self.swig_libdir, dir, file), out)
File "/h/46/james/lib/python2.4/shutil.py", line 82, in copy
copymode(src, dst)
File "/h/46/james/lib/python2.4/shutil.py", line 61, in copymode
os.chmod(dst, mode)
OSError: [Errno 2] No such file or directory:
'subversion/bindings/swig/proxy/runtime.swg'
make: *** [subversion/bindings/swig/proxy/swig_perl_external_runtime.swg]
Error 1
}}}
My patch fixes the problem, because it ensures that
external_runtime.py is only run once, and therefore is not run
multiple times in parallel.
Cheers,
David
P.S. (It's also possible to fix this issue by updating
external_runtime.py to generate each dependency, including the long
long fix and the checked-out SWIG headers, separately. To implement
this, we'd probably split up external_runtime.py into several scripts,
including a 'swig_checkout.py'. Would anyone like to work on this?)
--
David James -- http://www.cs.toronto.edu/~james
Re: Fix make -j for SWIG bindings
Posted by Philip Martin <ph...@codematters.co.uk>.
Max Bowsher <ma...@ukf.net> writes:
> David James wrote:
>> On 12/15/05, Philip Martin <ph...@codematters.co.uk> wrote:
>>
>>>A more annoying problem is that the 'make -j4 swig-py' parallel build
>>>is broken, it sometimes fails with
>>>
>>
>> Philip, does the attached patch solve the issue with make -j?
I haven't tried the patch.
>> [[[
>>
>> * build/generator/gen_make.py
>> Ensure that external_runtime.py is only called once. This fixes
>> "make -j", which crashes when multiple instances of
>> external_runtime.py try to create the same files.
>>
>> Found by: philip
>>
>> ]]]
>>
> Ouch! That's kind of inelegant. -0.9
>
> Exactly what is going on here?
external_runtime.py gets invoked three times with exactly the same
arguments because build-outputs.mk has three targets that invoke it
(swig_python_external_runtime.swg, swig_perl_external_runtime.swg,
swig_ruby_external_runtime.swg) and the swig-headers target depends on
all three.
That's a bit odd, and changing the timing by only invoking it once may
make the problem go away (or it may make the problem worse :), but I
don't think it's the real cause of the problem.
> Surely it is a bug in 'make -j' if it executes the same make stages in
> multiple threads of control?
Usually 'make -j' bugs are caused by missing dependencies. In this
case I suspect that the svn_client.c generated by swig from
svn_client.i should have a dependency on swig-headers.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Fix make -j for SWIG bindings (was: swig: srcdir vs. blddir)
Posted by Max Bowsher <ma...@ukf.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
David James wrote:
> On 12/15/05, Philip Martin <ph...@codematters.co.uk> wrote:
>
>>A more annoying problem is that the 'make -j4 swig-py' parallel build
>>is broken, it sometimes fails with
>>
>>/usr/local/swig-1.3.24/bin/swig -I/home/pm/sw/subversion/obj/../svn/subversion/include -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig/include -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig/proxy -I/home/pm/sw/subversion/obj/subversion/bindings/swig/proxy -I/usr/local/apache2/include -I/usr/local/apache2/include -python -classic -w451 -w305 -o ../svn/subversion/bindings/swig/python/svn_client.c ../svn/subversion/bindings/swig/svn_client.i
>>Traceback (most recent call last):
>> File "build/generator/swig/external_runtime.py", line 107, in ?
>> gen.write()
>> File "build/generator/swig/external_runtime.py", line 99, in write
>> self.write_long_long_fix()
>> File "build/generator/swig/external_runtime.py", line 82, in write_long_long_fix
>> self.checkout("python","python.swg")
>> File "/home/pm/sw/subversion/svn/build/generator/swig/__init__.py", line 56, in checkout
>> shutil.copy("%s/%s/%s" % (self.swig_libdir, dir, file), out)
>> File "/usr/lib/python2.3/shutil.py", line 73, in copy
>> copymode(src, dst)
>> File "/usr/lib/python2.3/shutil.py", line 52, in copymode
>> os.chmod(dst, mode)
>>OSError: [Errno 2] No such file or directory: 'subversion/bindings/swig/proxy/python.swg'
>
>
> Philip, does the attached patch solve the issue with make -j?
>
> [[[
>
> * build/generator/gen_make.py
> Ensure that external_runtime.py is only called once. This fixes
> "make -j", which crashes when multiple instances of
> external_runtime.py try to create the same files.
>
> Found by: philip
>
> ]]]
>
> Index: build/generator/gen_make.py
> ===================================================================
> --- build/generator/gen_make.py (revision 17814)
> +++ build/generator/gen_make.py (working copy)
> @@ -148,10 +148,16 @@
> fname = '%s/swig_%s_external_runtime.swg' % (self.swig.proxy_dir, lang)
> swig_runtime_fnames.append(fname)
> if not self.release_mode:
> + # external_runtime.py generates the SWIG runtimes for all languages
> + # the first time it is called
> + first_runtime = swig_runtime_fnames[0]
> + other_runtimes = " ".join(swig_runtime_fnames[1:])
> self.ofile.write(
> - '%s:\n' % " ".join(swig_runtime_fnames) +
> - '\tcd $(top_srcdir) && $(PYTHON)' +
> + '%s: %s\n'
> + '%s:\n'
> + '\tcd $(top_srcdir) && $(PYTHON)'
> ' build/generator/swig/external_runtime.py build.conf "$(SWIG)"\n\n'
> + % (other_runtimes, first_runtime, first_runtime)
> )
>
> self.ofile.write(
Ouch! That's kind of inelegant. -0.9
Exactly what is going on here?
Surely it is a bug in 'make -j' if it executes the same make stages in
multiple threads of control?
Max.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Cygwin)
iD8DBQFDo+0RfFNSmcDyxYARAsHtAJ9MJBFqoV2Xk7gctIsI4jCvT4vRlACglB4L
DFM2Zf0mqRfKlP8itRWx7eg=
=yu8z
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org