Re: Problems releasing 0.10.1 (nslater, back at you!)

[Paul Davis <pa...@gmail.com>]

On Fri, Nov 6, 2009 at 3:00 PM, Noah Slater <ns...@tumbolia.org> wrote:
> Getting this error trying to run make distcheck:
[snip]
> prove test/etap/*.t
> Cannot determine source for test/etap/*.t at
> /usr/share/perl/5.10/App/Prove.pm line 496

Noah,

Short answer: not reproducible
Long answer: I hate make distcheck.

I spent a whole bunch of time trying to get make distcheck to do the
tests properly. It frustrated me greatly. Then I threw something. Then
I stopped caring because you have check as a dependency of distsign.

distcheck causes lots of weirdness because of its VPATH builds. This
causes lots of exciting issues. Basically, the long and short of it is
that make check never worked properly via make distcheck. When I
refactored couchjs and added js tests I fought a long grueling battle
with distcheck before being defeated and writing this in the root
Makefile.am:

check:
    if test -f "test/etap/001-load.t"; then \
        prove test/etap/*.t; \
    fi

Granted the newer version also had some js test stuff, but that's the
basic idea.

The two options I see are: I'll commit and backport that check to only
run prove when check is being run from a not VPATH build, or you can
spend a month or two trying to bring all tests into comfortable
compatibility with distcheck.

HTH,
Paul Davis

Re: Problems releasing 0.10.1 (nslater, back at you!)

[Robert Newson <ro...@gmail.com>]

Just to break the monotony of the ping-pong session, I value the
effort Noah is exerting to make the build system this solid. I get
Paul's pragmatic approach too, but I personally side with doing the
build/autoconf thing right up front and head off weird build and
packaging horrors as early as possible.

/cranes neck back in

B.

On Sun, Nov 8, 2009 at 7:09 PM, Noah Slater <ns...@tumbolia.org> wrote:
>
> On 8 Nov 2009, at 18:45, Paul Davis wrote:
>
>> Who in the world actually does this?
>
> Ubuntu, Debian, etc. I'm sure there are many more.
>
> VPATH builds are commonplace where automation is used.
>
> End users don't usually use them, but we target more than just end users.
>
>> I would be much less whiney if I all of a sudden found out that we have
>> someone being uberawesome and running make distcheck on many many
>> platforms and as part of that setup used the VPATH builds.
>
> Forget distcheck, this is broken for check with a VPATH setup.
>
>> Of course its possible. Just check if you have source files in your
>> build directory.
>
> I cannot agree to a change that makes the test suite non-functional for
> VPATH builds.
>
>> I don't follow. I'm assuming that not lots of people use VPATH builds.
>
> Yes, I that is an unfounded assumption.
>
>> If there is some hidden build factory out there that's being kind
>> enough to build CouchDB on many platforms then I'd be orders of
>> magnitude more interested in making this work.
>
> Ubuntu, Debian, etc.
>
>> But the current trend I see is that it breaks distcheck.
>
> Forget distcheck, that is a symptom of check not working for VPATH setups.
>
>> To me, I'm not seeing the motivation
>> to do anything more than run the checks as part of distsign as opposed
>> to distcheck.
>
> You're treating the symptoms, and not the cause.
>
> The cause is that our test suite cannot handle VPATH settups.
>
> This is a bug.
>
>> I haven't been convinced that this is a wrong vs. right issue.
>
> We use an Autoconf build system, so we should support VPATH setups.
>
>> There is a lot of extra stuff that goes into VPATH builds and making sure
>> they work properly..
>
>> And its just added weight as we accumulate more
>> and more testing and build infrastructure. I just haven't been
>> convinced that there's a reason we should carry that.
>
> Because we're trying to grow our user base, not shrink it - and that means
> not cutting off deployment scenarios because it was a bit of a chore to get
> it working properly.
>
> This behaviour is not some new crazy feature I dreamt up one night, it's a
> standard, and well understood part of having an Autoconf build system -
> arguably the oldest active, and most popular open source build system that
> exists.
>
> And sure, it's tricky. But packaging is hard, welcome to packaging.
>

Re: Problems releasing 0.10.1 (nslater, back at you!)

[Paul Davis <pa...@gmail.com>]

Now that I've updated everything to run under make distcheck, it
occurs to me that its such a big patch that I'm not sure if it should
be backported or not. And even if it should be back ported, that's
gonna be a whole lot of fun as it definitely won't apply without
conflict. Not sure how hard it'd be to resolve, but it sure doesn't
sound like fun.

Yep.

Re: Problems releasing 0.10.1 (nslater, back at you!)

[Noah Slater <ns...@tumbolia.org>]

On 8 Nov 2009, at 20:24, Paul Davis wrote:

> Let me rephrase slightly, I assume very few people are dependent on
> running the test suite as part of  a VPATH build. For instance, not a
> single person has even contributed a "Yes, I use VPATH builds and
> would appreciate the test suite working under that setup." If its so
> common, where are these people?

I didn't say it was common, I said it was important.

>>> If there is some hidden build factory out there that's being kind
>>> enough to build CouchDB on many platforms then I'd be orders of
>>> magnitude more interested in making this work.
>>
>> Ubuntu, Debian, etc.
>
> There are CouchDB packages available for those distros already. And
> our test suite has never run as part of distcheck.

Sure, that our tests can't be run automatically for these distros is a  
huge bug.

>> This is a bug.
>
> Patches welcome :)

I offered a hacking session, and I am prepared to do the work myself  
even.

> We do. Building and installing CouchDB with a VPATH setup works just
> fine. I'm only looking to be convinced that all of the pain that's
> involved with supporting VPATH tests is worth it.

If you don't think it is worth your time, I will do it.

> Just to reiterate, the builds and packaging would work just dandy. We
> just wouldn't be able to run the tests during a VPATH build. We've
> survived this long without it.

Only because we didn't have any unit tests!

I'm not saying that lack of tests should prevent release, I'm saying  
that if we have tests, then they should work correctly in the fashion  
that our build system already advertises. However, I don't think this  
is necessarily release critical. I do consider a broken distcheck to  
be release critical, and will not, and cannot, release without it. If  
people want to ship 0.10.1 by disabling the unit tests so that this  
target doesn't fail, then I am reluctantly prepared to take that option.

> But I'm trying to understand why it's this way to begin with. What is
> the scenario that prevents people from putting the tarball on the
> read-only partition and expanding it to their $(top_builddir)?

This is largely orthogonal to our issues. We chose an Autotools  
system, and so it is best practice to design our build system around  
that, and the expectations people will have of that. If you want to  
understand the rationale behind it, I suggest we move this to the  
Autotools mailing lists, where I am sure they will have a plethora use  
cases to educate us with.

There are some here:

http://www.gnu.org/software/hello/manual/automake/VPATH-Builds.html

> It may appear that I'm just being a crotchety developer not wanting to
> go through and fix things, but my concerns are that anything we do in
> the test suite that touches a file will be thoroughly more complex
> because of this. Any file needs to be identified as "reading a source
> file", "reading a built file", or "needs to be writable". Every file.
> And we have to add the overhead to communicate $(top_srcdir) and
> $(top_builddir) to all parts of the code that touch those files. And
> then if we have static files that need that information, then they
> have to be built by configure or make as part of the build. And every
> person that wants to hack on those tests will need to understand all
> of those intricacies.

Yes, packaging is hard to get right.

> To me, that is a large weight for our test suite going forward.

No, it's not really.

A simple wrapper script that takes the build variables from configure  
should do it.

> Especially when our current suite is roughly one fourth (judging from
> coverage) of what it should be. Trying to recruit people beyond the
> three or so that have contributed to those tests is just going to get
> harder the more complicated that stuff gets.

This is pure supposition, and I'm not convinced it is well founded.

> So yes, it would be nice to have VPATH builds be able to run the
> tests. But it would also be nice to have some idea as to the merit
> behind carrying this forward.

Please read this:

http://www.gnu.org/software/hello/manual/automake/VPATH-Builds.html

Packaging is exceptionally hard to get right, and Autotools is the  
culmination of decades of experience packaging and building software  
so that it works on the largest possible number of systems. Think of  
it has executable packaging lore. Hehe.

"Here’s the basic problem: you’re writing a text editor. Stop doing  
that. It’s 2007. Saying to yourself “I’m gonna build my own text  
editor” is as silly as saying “I’m gonna build my own build system” or  
“I’m gonna build my own amusement park.” Blackjack and hookers and all  
that. Writing a great text editor is insanely difficult. There is a  
certain class of software that sounds easy but is actually insanely  
difficult. I call it “garden path software.” If I ever start a  
software company, I’ll name it “Garden Path Software,” but until then,  
just stop."

- http://diveintomark.org/archives/2007/01/21/wrongroom

> Either way, I've pretty much resigned myself to getting these to run
> as part of the VPATH build. The autotools juggernaut has successfully
> claimed yet another piece of my soul.

Like I said, it's painful - but our goal is ubiquity, and the path to  
ubiquity is painful.

GO GO GADGET RHETORIC!

Over and out,

Noah

Re: Problems releasing 0.10.1 (nslater, back at you!)

[Paul Davis <pa...@gmail.com>]

>> I don't follow. I'm assuming that not lots of people use VPATH builds.
>
> Yes, I that is an unfounded assumption.

Let me rephrase slightly, I assume very few people are dependent on
running the test suite as part of  a VPATH build. For instance, not a
single person has even contributed a "Yes, I use VPATH builds and
would appreciate the test suite working under that setup." If its so
common, where are these people?

>> If there is some hidden build factory out there that's being kind
>> enough to build CouchDB on many platforms then I'd be orders of
>> magnitude more interested in making this work.
>
> Ubuntu, Debian, etc.

There are CouchDB packages available for those distros already. And
our test suite has never run as part of distcheck.

>> To me, I'm not seeing the motivation
>> to do anything more than run the checks as part of distsign as opposed
>> to distcheck.
>
> You're treating the symptoms, and not the cause.
>
> The cause is that our test suite cannot handle VPATH settups.
>
> This is a bug.

Patches welcome :)

>> I haven't been convinced that this is a wrong vs. right issue.
>
> We use an Autoconf build system, so we should support VPATH setups.

We do. Building and installing CouchDB with a VPATH setup works just
fine. I'm only looking to be convinced that all of the pain that's
involved with supporting VPATH tests is worth it.

>> There is a lot of extra stuff that goes into VPATH builds and making sure
>> they work properly..
>
>> And its just added weight as we accumulate more
>> and more testing and build infrastructure. I just haven't been
>> convinced that there's a reason we should carry that.
>
> Because we're trying to grow our user base, not shrink it - and that means
> not cutting off deployment scenarios because it was a bit of a chore to get
> it working properly.

Just to reiterate, the builds and packaging would work just dandy. We
just wouldn't be able to run the tests during a VPATH build. We've
survived this long without it.

> This behaviour is not some new crazy feature I dreamt up one night, it's a
> standard, and well understood part of having an Autoconf build system -
> arguably the oldest active, and most popular open source build system that
> exists.

The argument that its part of "The Autotools Way" is not lost on me.
But I'm trying to understand why it's this way to begin with. What is
the scenario that prevents people from putting the tarball on the
read-only partition and expanding it to their $(top_builddir)?

> And sure, it's tricky. But packaging is hard, welcome to packaging.

It may appear that I'm just being a crotchety developer not wanting to
go through and fix things, but my concerns are that anything we do in
the test suite that touches a file will be thoroughly more complex
because of this. Any file needs to be identified as "reading a source
file", "reading a built file", or "needs to be writable". Every file.
And we have to add the overhead to communicate $(top_srcdir) and
$(top_builddir) to all parts of the code that touch those files. And
then if we have static files that need that information, then they
have to be built by configure or make as part of the build. And every
person that wants to hack on those tests will need to understand all
of those intricacies.

To me, that is a large weight for our test suite going forward.
Especially when our current suite is roughly one fourth (judging from
coverage) of what it should be. Trying to recruit people beyond the
three or so that have contributed to those tests is just going to get
harder the more complicated that stuff gets.

So yes, it would be nice to have VPATH builds be able to run the
tests. But it would also be nice to have some idea as to the merit
behind carrying this forward.

Either way, I've pretty much resigned myself to getting these to run
as part of the VPATH build. The autotools juggernaut has successfully
claimed yet another piece of my soul.

HTH,
Paul Davis

Re: Problems releasing 0.10.1 (nslater, back at you!)

[Noah Slater <ns...@tumbolia.org>]

On 8 Nov 2009, at 18:45, Paul Davis wrote:

> Who in the world actually does this?

Ubuntu, Debian, etc. I'm sure there are many more.

VPATH builds are commonplace where automation is used.

End users don't usually use them, but we target more than just end  
users.

> I would be much less whiney if I all of a sudden found out that we  
> have
> someone being uberawesome and running make distcheck on many many
> platforms and as part of that setup used the VPATH builds.

Forget distcheck, this is broken for check with a VPATH setup.

> Of course its possible. Just check if you have source files in your
> build directory.

I cannot agree to a change that makes the test suite non-functional  
for VPATH builds.

> I don't follow. I'm assuming that not lots of people use VPATH builds.

Yes, I that is an unfounded assumption.

> If there is some hidden build factory out there that's being kind
> enough to build CouchDB on many platforms then I'd be orders of
> magnitude more interested in making this work.

Ubuntu, Debian, etc.

> But the current trend I see is that it breaks distcheck.

Forget distcheck, that is a symptom of check not working for VPATH  
setups.

> To me, I'm not seeing the motivation
> to do anything more than run the checks as part of distsign as opposed
> to distcheck.

You're treating the symptoms, and not the cause.

The cause is that our test suite cannot handle VPATH settups.

This is a bug.

> I haven't been convinced that this is a wrong vs. right issue.

We use an Autoconf build system, so we should support VPATH setups.

> There is a lot of extra stuff that goes into VPATH builds and making  
> sure
> they work properly..

> And its just added weight as we accumulate more
> and more testing and build infrastructure. I just haven't been
> convinced that there's a reason we should carry that.

Because we're trying to grow our user base, not shrink it - and that  
means not cutting off deployment scenarios because it was a bit of a  
chore to get it working properly.

This behaviour is not some new crazy feature I dreamt up one night,  
it's a standard, and well understood part of having an Autoconf build  
system - arguably the oldest active, and most popular open source  
build system that exists.

And sure, it's tricky. But packaging is hard, welcome to packaging.

Re: Problems releasing 0.10.1 (nslater, back at you!)

[Paul Davis <pa...@gmail.com>]

On Sun, Nov 8, 2009 at 8:36 AM, Noah Slater <ns...@tumbolia.org> wrote:
>
> On 8 Nov 2009, at 04:20, Paul Davis wrote:
>
>> Well, the lack of distcheck actually running check hasn't appeared to
>> cause any sort of issue thus far.
>
> Apart from the build not running the tests, which is antithetical to the
> point of having them.
>

Well, our build doesn't run them. Part of the release distribution
preparation tries to run them though. Granted, until recently,
distcheck never tried to run them, they're only listed as a dependency
of the distsign target which doesn't suffer from VPATH awesome.

>> Teach all test code about the difference between $(top_srcdir) and
>> $(top_builddir) or punt and just run the tests as part of distsign
>> when the test code runs with a unified filesystem view.
>
> Yes, the tests should know how to deal with VPATH builds.
>
>> I could be bothered to try and futz around with getting the tests to
>> run under distcheck, but I worry that the overheard is gonna start to
>> annoy anyone that feels like adding to them.
>
> Let's get it working first, and then discuss ways of making it easier,
>
>> Its fairly arduous considering the use case is for people that want to
>> ./configure with a
>> build path outside of $(top_srcdir). And by that, I mean people that
>> want to run the test suite in that scenario.
>
> Implicit is the suggestion that nobody actually does this.

Well, lemme make it explicit: Who in the world actually does this? I
would be much less whiney if I all of a sudden found out that we have
someone being uberawesome and running make distcheck on many many
platforms and as part of that setup used the VPATH builds. In that
case, yes, I would be more willing to make sure that this all worked
so that we could have reliance on builds working. But, alas, I know of
no such setup.

>> Not running the tests
>> when we detect the VPATH build to me is AOK, as anyone doing the split
>> build can check separately.
>
> Your assuming that this is possible.

Of course its possible. Just check if you have source files in your
build directory.

> Think about the scenario where a maintainer uploads a new version of the
> CouchDB package for a distribution, and that package is then built
> automatically on a number of architectures, either for, or by the end users.
> In the situation where these builds require a VPATH (say, because the
> sources are mounted on a read-only filesystem) then they will be unable to
> install CouchDB.

The package would build and install just fine. If not, then well, it
would already be broken. The only thing they couldn't do is run the
Erlang test suite as part of that build. Or, the ./utils/run server
for that matter as it doesn't respect VPATH builds either (though its
an easy fix).

>> Granted, if lots of people all of a sudden reply to this with a "Hey,
>> its absolutely necessary to run the test suite from the weird builds
>> we're doing" then I'll give it more weight, but I just don't see this
>> as very productive in the short, mid, or long term.
>
> You're assuming an awful lot about how people build packages in the wild.

I don't follow. I'm assuming that not lots of people use VPATH builds.
If there is some hidden build factory out there that's being kind
enough to build CouchDB on many platforms then I'd be orders of
magnitude more interested in making this work. But the current trend I
see is that it breaks distcheck. To me, I'm not seeing the motivation
to do anything more than run the checks as part of distsign as opposed
to distcheck.

>> Or in other words, "Fixing the build" could be "not running tests on
>> distcheck" or "fixing all tests to respect VPATH builds" one is
>> simple, the other is not.
>
> And one is broken, and the other is not.
>

I haven't been convinced that this is a wrong vs. right issue. There
is a lot of extra stuff that goes into VPATH builds and making sure
they work properly. And its just added weight as we accumulate more
and more testing and build infrastructure. I just haven't been
convinced that there's a reason we should carry that.

Paul Davis

Re: Problems releasing 0.10.1 (nslater, back at you!)

[Noah Slater <ns...@tumbolia.org>]

On 8 Nov 2009, at 04:20, Paul Davis wrote:

> Well, the lack of distcheck actually running check hasn't appeared to
> cause any sort of issue thus far.

Apart from the build not running the tests, which is antithetical to  
the point of having them.

> Teach all test code about the difference between $(top_srcdir) and
> $(top_builddir) or punt and just run the tests as part of distsign
> when the test code runs with a unified filesystem view.

Yes, the tests should know how to deal with VPATH builds.

> I could be bothered to try and futz around with getting the tests to
> run under distcheck, but I worry that the overheard is gonna start to
> annoy anyone that feels like adding to them.

Let's get it working first, and then discuss ways of making it easier,

> Its fairly arduous considering the use case is for people that want  
> to ./configure with a
> build path outside of $(top_srcdir). And by that, I mean people that
> want to run the test suite in that scenario.

Implicit is the suggestion that nobody actually does this.

> Not running the tests
> when we detect the VPATH build to me is AOK, as anyone doing the split
> build can check separately.

Your assuming that this is possible.

Think about the scenario where a maintainer uploads a new version of  
the CouchDB package for a distribution, and that package is then built  
automatically on a number of architectures, either for, or by the end  
users. In the situation where these builds require a VPATH (say,  
because the sources are mounted on a read-only filesystem) then they  
will be unable to install CouchDB.

> Granted, if lots of people all of a sudden reply to this with a "Hey,
> its absolutely necessary to run the test suite from the weird builds
> we're doing" then I'll give it more weight, but I just don't see this
> as very productive in the short, mid, or long term.

You're assuming an awful lot about how people build packages in the  
wild.

> Or in other words, "Fixing the build" could be "not running tests on
> distcheck" or "fixing all tests to respect VPATH builds" one is
> simple, the other is not.

And one is broken, and the other is not.

Re: Problems releasing 0.10.1 (nslater, back at you!)

[Paul Davis <pa...@gmail.com>]

On Sat, Nov 7, 2009 at 8:46 PM, Noah Slater <ns...@tumbolia.org> wrote:
>
> On 6 Nov 2009, at 20:52, Paul Davis wrote:
>
>> Also, I would just like to mention that its worked fine so far. ;) And
>> to the one guy in the shack on an island using a weird VPATH build
>> that wants to run the unit tests: Patches welcome.
>
> All I have in this world is my balls and my release procedure, and I don't
> break 'em for no one. Or in other words, if we want this to work with
> Debian, Ubuntu, Gentoo, etc, this needs to work. No questions.  Or if you do
> have questions, I will just quote more Scarface at you.
>
>

Well, the lack of distcheck actually running check hasn't appeared to
cause any sort of issue thus far. In the end it boils down to this:

Teach all test code about the difference between $(top_srcdir) and
$(top_builddir) or punt and just run the tests as part of distsign
when the test code runs with a unified filesystem view.

I could be bothered to try and futz around with getting the tests to
run under distcheck, but I worry that the overheard is gonna start to
annoy anyone that feels like adding to them. Its fairly arduous
considering the use case is for people that want to ./configure with a
build path outside of $(top_srcdir). And by that, I mean people that
want to run the test suite in that scenario. Not running the tests
when we detect the VPATH build to me is AOK, as anyone doing the split
build can check separately.

Granted, if lots of people all of a sudden reply to this with a "Hey,
its absolutely necessary to run the test suite from the weird builds
we're doing" then I'll give it more weight, but I just don't see this
as very productive in the short, mid, or long term.

Or in other words, "Fixing the build" could be "not running tests on
distcheck" or "fixing all tests to respect VPATH builds" one is
simple, the other is not.

HTH,
Paul Davis

Re: Problems releasing 0.10.1 (nslater, back at you!)

[Noah Slater <ns...@tumbolia.org>]

On 6 Nov 2009, at 20:52, Paul Davis wrote:

> Also, I would just like to mention that its worked fine so far. ;) And
> to the one guy in the shack on an island using a weird VPATH build
> that wants to run the unit tests: Patches welcome.

All I have in this world is my balls and my release procedure, and I  
don't break 'em for no one. Or in other words, if we want this to work  
with Debian, Ubuntu, Gentoo, etc, this needs to work. No questions.   
Or if you do have questions, I will just quote more Scarface at you.

Re: Problems releasing 0.10.1 (nslater, back at you!)

[Paul Davis <pa...@gmail.com>]

On Fri, Nov 6, 2009 at 3:33 PM, Noah Slater <ns...@tumbolia.org> wrote:
>
> On 6 Nov 2009, at 20:27, Paul Davis wrote:
>
>> distcheck causes lots of weirdness because of its VPATH builds.
>
> Painful, I know, but only because it tests every type of install
> configuration.
>
> If distcheck fails, someone somewhere, will have a broken build.
>
>> The two options I see are: I'll commit and backport that check to only
>> run prove when check is being run from a not VPATH build, or you can
>> spend a month or two trying to bring all tests into comfortable
>> compatibility with distcheck.
>
> Let's tag team on this. You free Sunday for a hack?
>
>

Its a possibility. I've gotten most of the weird bits fixed with
utils/run and some other places for this to actually work. I got all
the way to noticing that etap's were using a path like
"test/etap/file.tmp" as a temporary file to write to. I didn't feel
like fixing that at the time so I punted.

Also, I would just like to mention that its worked fine so far. ;) And
to the one guy in the shack on an island using a weird VPATH build
that wants to run the unit tests: Patches welcome.

Yup.

Paul Davis

Re: Problems releasing 0.10.1 (nslater, back at you!)

[Noah Slater <ns...@tumbolia.org>]

On 6 Nov 2009, at 20:27, Paul Davis wrote:

> distcheck causes lots of weirdness because of its VPATH builds.

Painful, I know, but only because it tests every type of install  
configuration.

If distcheck fails, someone somewhere, will have a broken build.

> The two options I see are: I'll commit and backport that check to only
> run prove when check is being run from a not VPATH build, or you can
> spend a month or two trying to bring all tests into comfortable
> compatibility with distcheck.

Let's tag team on this. You free Sunday for a hack?