You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Greg Stein <gs...@lyra.org> on 2001/05/24 14:20:55 UTC
new build system (was: Re: CVS update: MODIFIED: ac-helpers ...)
On Thu, May 24, 2001 at 01:40:17PM -0000, gstein@tigris.org wrote:
> User: gstein
> Date: 01/05/24 06:40:17
>
> Modified: ac-helpers .cvsignore svn-apache.m4
> Added: . Makefile.in
> Log:
> Switch over to the new non-recursive build system.
>...
Okay... this is it. We're now on the build system.
"It works on my machine."
I suspect there may be some tweaks to make on differents OSs. I'd be
interested to hear if Ben can really build with normal BSD make. It should
be possible.
The code supports building, installation, checking, and dependencies. It
does *NOT* yet deal with the doc/ subdirectory. That is next; I figured this
could be rolled out and get the kinks worked out while I do the doc/ stuff.
Oh, it doesn't build Neon or APR yet either. I also saw a problem where
libsvn_fs wasn't getting built before linking one of the test proggies (see
below).
Basic operation: same as before.
$ ./autogen.sh
$ ./configure OPTIONS
$ make
$ make check
$ make install
There are some "make check" scripts that need to be fixed up. That'll happen
RSN. Some of them create their own log, rather than spewing to stdout (where
the top-level make will place the output into [TOP]/tests.log).
The old Makefile.am files are still around, but I'll be tossing those along
with a bunch of tweaks to all the .cvsignore files. There are a few other
cleanups, too. But that can happen as a step two.
[ $ cvs rm -f `find . -name Makefile.rm`
See the mistake in that line? I didn't when I typed it. The find returned
nothing, so cvs rm -f proceeded to delete my entire tree. And the -f made
sure to delete all my source files, too. Good fugging thing that I had my
mods in some Emacs buffers, or I'd be bitching.
I am *so* glad that Ben coded SVN to *not* delete locally modified files
*and* that we have an "undel" command. I had to go and tweak a bazillion
Entries files to undo the delete...
]
The top-level make has a number of shortcuts in it (well, actually in
build-outputs.mk):
$ make subversion/libsvn_fs/libsvn_fs.la
or
$ make libsvn_fs
The two are the same. So... when your test proggie fails to link because
libsvn_fs isn't around, just run "make libsvn_fs" to build it immediately,
then go back to the regular "make".
Note that the system still conditionally builds the FS stuff based on
whether DB is available, and mod_dav_svn if Apache is available.
Handy hint: if you don't like dependencies, then you can do:
$ ./autogen.sh -s
That will skip the dependency generation that goes into build-outputs.mk. It
makes the script run quite a bit faster (48 secs vs 2 secs on my poor little
Pentium 120).
Note that if you change build.conf, you can simply run:
$ ./gen-make.py build.conf
to regen build-outputs.mk. You don't have to go back through the whole
autogen.sh / configure process.
You should also note that autogen.sh and configure run much faster now that
we don't have the automake crap. Oh, and our makefiles never re-run
configure on you out of the blue (gawd, I hated when automake did that to
me).
Obviously, there are going to be some tweaky things going on. I also think
that the "shadow" builds or whatever they're called (different source and
build dirs) are totally broken. Something tweaky will have to happen there.
But, thankfully, we only have one Makefile to deal with.
Note that I arrange things so that we have one generated file
(build-outputs.mk), and one autoconf-generated file (Makefile from .in).
I also tried to shove as much logic/rules into Makefile.in. Keeping
build-outputs.mk devoid of rules (thus, implying gen-make.py devoid of rules
in its output generation) manes that tweaking rules in Makefile.in is much
more approachable to people.
I think that is about it. Send problems to the dev@ list and/or feel free to
dig in and fix them yourself. My next steps are mostly cleanup. After that,
I'm going to toss out our use of libtool and rely on APR's libtool setup (no
need for us to replicate what APR already did).
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: new build system (was: Re: CVS update: MODIFIED: ac-helpers ...)
Posted by Karl Fogel <kf...@collab.net>.
RADICS Peter <mi...@lbcons.net> writes:
> Could you explain some rationale behind this rewamping of the make
> process? It's not that I'm against or something, I'm just curious.
> In fact, I also had my days (bad days, that is) with automake,
> but I thought the next release of automake would solve most of the
> problems with the current system (according to The Book on automake
> and friends), but now I'm interested why this project chose to go
> in another direction.
> (I'm trying to learn from other ppls mistakes/experiences here :)
(Greg may want to post a more detailed answer).
We've had a lot of inter-library circular dependency problems with the
recursive Makefile system, that required us to spend unexpected
amounts of time coordinating build order between directories.
See Peter Miller's paper "Recursive Make Considered Harmful":
http://www.pcug.org.au/~millerp/rmch/recu-make-cons-harm.html
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: new build system
Posted by "Chris D. Sloan" <cd...@cs.hmc.edu>.
On Fri, May 25, 2001 at 10:31:56AM -0400, Greg Hudson wrote:
> Moreover, having a single Makefile is less functional in that you
> can't run "make" in a subdirectory to build only targets in that
> subdirectory, as is often convenient. (And it's not just convenient
> to save runtime; it's convenient because (1) the world isn't an
> optimal place, and sometimes the build blows out in a directory you
> didn't care about, and it's easier to just build subcomponents than to
> fix the problem, and (2) you're often already in the subdir in a
> previously completed build tree, making some change, and don't want to
> jump back to the top level to test it.)
With recursive make, you can type make in a subdirectory and get just
that directory, and I agree that's useful, but when you are at the top
directory you cannot make specific targets in the recursive makefiles.
One makefile basically ignores directories, but recursive makefiles
artificially hide their contents. Personally, I have had much better
luck with a single makefile than with recursive makefiles, but a fair
amount probably depends on personal style.
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: new build system
Posted by Greg Stein <gs...@lyra.org>.
On Sat, Jun 02, 2001 at 10:35:17AM -0400, Greg Hudson wrote:
> Two more notes:
>
> >> Why doesn't apr's configure script avoid touching apr.h when there
> >> is no actual change?
>
> > AC_OUTPUT() doesn't do that, AFAIK.
>
> AC_CONFIG_HEADER does. From experience, I believe it would be
> possible to use AC_CONFIG_HEADER with apr.h without corrupting the
> namespace with autoconf symbols.
Hmm. I'll take a look at that. It would be nice not to touch that header on
a simple reconfig.
> (Even better would be to make apr.h
> not vary per platform and per compiler, but that's asking a lot for a
> portability layer.)
The various type mappings are in there, so it will have to change.
> > What I find is pretty neat is when you do a "make" now, when
> > everything is already up-to-date. We get a bazillion lines of output
> > from APR as it recurses all over the place, telling you "entering
> > directory X" and "leaving directory X". Then our portion prints
> > nothing. Nice and quiet :-)
>
> If you don't like those enter/exit messages from gnu make, just use
> make -S.
Ah. Cool. Dang... where do you find this stuff? :-) (and don't say RTFM)
I also realized that SVN users can do "make local-all" to just build the SVN
portions and skip APR/Neon. Once you build the two external projects, the
local-all trick can keep things much quieter.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: new build system
Posted by Greg Stein <gs...@lyra.org>.
On Fri, May 25, 2001 at 10:31:56AM -0400, Greg Hudson wrote:
>...
> > On that last bit: GCC's dependencies included every file included,
> > not just the SVN files.
>
> (Your statement is too expansive; the dependencies shouldn't include
> system headers.) How is it supposed to know that the APR files are
> part of a different project?
At the moment, we only include specific headers into dependencies. Namely,
those in the "current directory" and the headers listed in the [includes]
section of build.conf.
> How do you know that a change to apr.h
> doesn't affect the files which include it?
I don't :-) I took the position that we simply assume changes there will
not affect us. Given that APR is supposed to be a library (someday :-), then
it will take pains to avoid binary incompatibility from one version to the
next.
> Why doesn't apr's
> configure script avoid touching apr.h when there is no actual change?
AC_OUTPUT() doesn't do that, AFAIK.
>...
> > shouldn't ever try it. But you can't change that behavior in
> > Automake.
>
> Right with you up until the last line; there's an AM_MAINTAINER_MODE
> option to change that behavior.
Ah. Okee... Thanks.
> > There were some things that we wanted to do which were a giant pain
> > when dealing with those scattered Makefiles. For example: adding the
> > --silent flag to every libtool line.
>
> I'm not sure why Subversion should be different from pretty much all
> other libtool-using programs in regards to the --silent flag, but I'm
> also not sure what the problem is; we seemed to be doing this before
> the build system change via the definition of LIBTOOL.
We had to redefine LIBTOOL in every Makefile.am. I called that "painful" :-)
The --silent flag is handy, preventing a lot of output from spewing during
the build process. And I've never needed the stuff that libtool prints out
(i.e. who cares what it does under the covers; it is just a tool).
> > Or a rule for creating the .la-a files needs when we do a
> > static-link install of mod_dav_svn into an Apache source tree.
>
> This problem is over my head, but I can certainly see how changing the
> object generation rules would be a pain in automake. I'm not actually
> a big fan of automake and don't use it in my own projects, but I do
> use recursive make, and have never seen a compelling reason not to.
Recursive make would be an option, and we could use a system similar to
APR(UTIL) where every Makefile includes a "rules.mk" file. It actually works
reasonably well. But we still have that whole recursion thing going on.
What I find is pretty neat is when you do a "make" now, when everything is
already up-to-date. We get a bazillion lines of output from APR as it
recurses all over the place, telling you "entering directory X" and "leaving
directory X". Then our portion prints nothing. Nice and quiet :-)
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: new build system
Posted by Joe Orton <jo...@btconnect.com>.
On Fri, May 25, 2001 at 10:31:56AM -0400, Greg Hudson wrote:
...
> > There were some things that we wanted to do which were a giant pain
> > when dealing with those scattered Makefiles. For example: adding the
> > --silent flag to every libtool line.
>
> I'm not sure why Subversion should be different from pretty much all
> other libtool-using programs in regards to the --silent flag, but I'm
> also not sure what the problem is; we seemed to be doing this before
> the build system change via the definition of LIBTOOL.
This was my mistake - it turns out you can add --silent to the LIBTOOL
variable everywhere by doing LIBTOOL="$LIBTOOL --silent" in configure.in
after using the PROG_LIBTOOL macro, so adding it to every Makefile.am
was overkill. :(
joe
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: new build system
Posted by Greg Stein <gs...@lyra.org>.
On Sun, May 27, 2001 at 12:29:36AM -0400, Garance A Drosihn wrote:
>...
> >Then there is that whole recursion thing I mentioned. We get
> >speed and clarity with the new system. Removing automake and
> >its bazillion substitutions has also dramatically sped up
> >autogen.sh and the last step of ./configure.
>
> Speeding up the build process also sounds like a win. As long
> as it all works right, this seems like a worthwhile alternative
> to try out. If it works, then faster builds are always nice
> to have!
If it didn't work, I wouldn't have checked it in :-)
> (any idea how well this works with 'make -j'?)
Actually, make -j should work much better with the single makefile. It can
parallelize the entire build, not just the bits within a single directory.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: new build system
Posted by Garance A Drosihn <dr...@rpi.edu>.
At 8:51 PM -0700 5/24/01, Greg Stein wrote:
>Automake also produces Makefiles that require GNU make. That
>is a big pain for the FreeBSD users out there, and some other
>fringe/older OSes (I've got to imagine that SunOS or AIX
>users will be happier).
As I am primarily a freebsd/openbsd/MacOS-10 user, this sounds
good to me... :-)
>The automake system also required GCC to be present to deal
>with dependency generation. And you couldn't turn that off!
>I have no idea how people were building SVN on AIX systems
>using the xlc compiler (maybe nobody tried?).
Chances are that anyone who did try has been forced to build
gcc/gmake due to other automake'd packages on the same systems.
I haven't tried to build SVN anywhere yet, but I do know that
our AIX and IRIX boxes do have gcc and gmake built for them.
>Then there is that whole recursion thing I mentioned. We get
>speed and clarity with the new system. Removing automake and
>its bazillion substitutions has also dramatically sped up
>autogen.sh and the last step of ./configure.
Speeding up the build process also sounds like a win. As long
as it all works right, this seems like a worthwhile alternative
to try out. If it works, then faster builds are always nice
to have!
(any idea how well this works with 'make -j'?)
--
Garance Alistair Drosehn = gad@eclipse.acs.rpi.edu
Senior Systems Programmer or gad@freebsd.org
Rensselaer Polytechnic Institute or drosih@rpi.edu
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: new build system
Posted by Greg Stein <gs...@lyra.org>.
On Thu, May 24, 2001 at 09:11:43PM +0200, RADICS Peter wrote:
> Hello gstein,
>
> Could you explain some rationale behind this rewamping of the make
> process?
No.
Just kidding... :-) As Karl pointed out, the paper on "Recursive Make
Considered Harmful" is actually quite a valid paper. The new build system is
a lot faster than before. We also have just a single Makefile to maintain,
rather than two dozen scattered across the system.
There were some things that we wanted to do which were a giant pain when
dealing with those scattered Makefiles. For example: adding the --silent
flag to every libtool line. Or a rule for creating the .la-a files needs
when we do a static-link install of mod_dav_svn into an Apache source tree.
Automake also produces Makefiles that require GNU make. That is a big pain
for the FreeBSD users out there, and some other fringe/older OSes (I've got
to imagine that SunOS or AIX users will be happier).
The automake system also required GCC to be present to deal with dependency
generation. And you couldn't turn that off! I have no idea how people were
building SVN on AIX systems using the xlc compiler (maybe nobody tried?).
The new dependency generation is portable, and also much more finely tuned.
On that last bit: GCC's dependencies included every file included, not just
the SVN files. This *really* pissed me off. Whenever I would go and
configure APR, it would touch include/apr.h. BAM! My entire Subversion
project would recompile. I've been lazy about moving to my faster
development machine, so that is actually quite painful. I've gotten pretty
used to avoiding unnecessary updates to APR. That's just wrong. The tools
should work *for* me, rather than against me. We want dependencies on the
stuff that we change, and use the presumption that our building blocks
(Neon, APR, etc) only change their APIs in compatible ways.
Another thing about automake is its insane predilection for rebuilding the
./configure script and associated files. Any change to those files, and
everything goes to seed. I think it is an absolute horror for a Makefile to
ever run ./confiure, for any reason. That is a user operation, not something
to be left to a Makefile. Same thing for generating ./configure from the
configure.in file -- we have autogen.sh to do that; the Makefile shouldn't
ever try it. But you can't change that behavior in Automake.
Automake also has some weird dependencies on a ChangeLog file being present.
That is silly. We might *distribute* one, but in day-to-day development?
Screw that.
The underlying point is that Automake may be fine for FSF/GNU projects,
which all adhere to a very specific set of rules. But for your Average Joe
project, we have our own policies and desires. Automake isn't flexible
enough for that.
Then there is that whole recursion thing I mentioned. We get speed and
clarity with the new system. Removing automake and its bazillion
substitutions has also dramatically sped up autogen.sh and the last step of
./configure.
Lastly, we have a lot better control over all the details of building,
installation, cleaning, etc. In particular, the "make check" stuff is a big
win. We were replacing that in all of the Makefiles for the test
directories. But they were all inconsistent. And they didn't work together.
etc etc.
It also just feels simpler. Need to change something? Go and tweak
Makefile.in and it fixes the whole system. Or maybe adjust gen-make.py to
alter how it produces build-outputs.mk.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: new build system (was: Re: CVS update: MODIFIED: ac-helpers ...)
Posted by RADICS Peter <mi...@lbcons.net>.
Hello gstein,
Could you explain some rationale behind this rewamping of the make
process? It's not that I'm against or something, I'm just curious.
In fact, I also had my days (bad days, that is) with automake,
but I thought the next release of automake would solve most of the
problems with the current system (according to The Book on automake
and friends), but now I'm interested why this project chose to go
in another direction.
(I'm trying to learn from other ppls mistakes/experiences here :)
Thanks for the info,
mitch
--
// RADICS Peter <mi...@lbcons.net> (http://lbcons.net)
//
// "If human beings don't keep exercising their lips,
// their brains start working." -- Ford Prefect
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org