You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Mark Hamstra <ma...@clearstorydata.com> on 2013/09/12 00:04:08 UTC
Maven SCM trouble
I'm in the process of trying to build Apache-fied Spark into our stack at
ClearStory. We've been using Maven to do Debian packaging as found in the
repl-bin module for quite a while, but that doesn't work now. To see the
failure, you need to build repl-bin with the `deb` profile -- e.g. "mvn
-Prepl-bin,deb install" from the project directory. The problem is with
the buildnumber-maven-plugin, which is used to attach the 8-digit short SHA
of the current git commit to the Debian package -- so, e.g.,
spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber plugin relies upon
the <scm> tag to determine which source code management system should be
used to lookup the version number.
That all worked great until the following was added to spark/pom.xml:
<parent>
<groupId>org.apache</groupId>
<artifactId>apache</artifactId>
<version>11</version>
</parent>
The problem is that the parent pom that the above includes has this within
it:
<scm>
<connection>scm:svn:
http://svn.apache.org/repos/asf/maven/pom/tags/apache-11</connection
<developerConnection>scm:svn:
https://svn.apache.org/repos/asf/maven/pom/tags/apache-11
</developerConnection>
<url>http://svn.apache.org/viewvc/maven/pom/tags/apache-11</url>
</scm>
And that essentially overrides the <scm> section of spark/pom.xml, which
means that the buildnumber plugin thinks that we are using SVN instead of
git, and the SHA lookup fails.
So, that is one known problem in the Maven build. I'm not at all certain
that there aren't others or lurking problems because of different versions
of dependencies, plugins, etc. being included by that org.apache:apache
artifact and also specified within the Spark pom files.
I'm pretty sure that we can't not include the equivalent of
org.apache:apache:11, since there is a lot of important stuff in there to
do Apache release management, etc. So, is there another, equivalent
artifact that we can use that doesn't confuse the <scm> issue, or some
other way around this problem? It looks like there are some other
git-specific plugins out there that we may be able to use instead of
buildnumber-maven-plugin, so that may be a valid and necessary solution to
the Debian packaging problem.
Can someone who knows Maven better than I take a look at the overlap
between org.apache:apache and the Spark pom files to see if there are
problems lurking there? Even if there aren't problems other than with the
buildnumber plugin, we may be able to trim a lot out of the Spark poms that
is already present in the apache parent pom.
Re: Maven SCM trouble
Posted by Henry Saputra <he...@gmail.com>.
I was about to suggest that. Thanks Patrick.
- Henry
On Wednesday, September 11, 2013, Patrick Wendell wrote:
> Ya I amended this for pushing the releases to maven. I actually
> changed that later to a different URI because it was incorrect (should
> have been https://)
>
> On Wed, Sep 11, 2013 at 11:41 PM, Gary Struthers
> <ag...@earthlink.net> wrote:
> > It looks like scm in the pom is only used to upload artifacts to Maven
> Central and isn't needed for building. Explained here:
> > http://books.sonatype.com/nexus-book/reference/_introduction_7.html
> >
> > If you run into a transitive dependency problem, there is a nice way to
> exclude them. Explained here:
> >
> http://books.sonatype.com/mvnref-book/reference/pom-relationships-sect-project-dependencies.html#pom-relationships-sect-conflict
> >
> > By far the best Maven documentation is in Sonatype's free books. They're
> accurate and up to date, unlike the Apache Maven docs or StackOverflow.
> > http://www.sonatype.com/resources/books
> >
> > Gary Struthers
> >
> > On Sep 11, 2013, at 11:21 PM, Mark Hamstra <ma...@clearstorydata.com>
> wrote:
> >
> >> Ah hah! Patrick figured it out:
> >>
> https://github.com/apache/incubator-spark/commit/905edf59db662868f55525118131cf102d366587
> .
> >> The buildnumber plugin and Debian packaging works correctly again.
> >> Thanks! I'll go through the poms tomorrow and see if there are any
> other
> >> such incompletely overriden elements that could be causing us troubles.
> >>
> >> Right now I am seeing two problems with the maven build: 1) I just had a
> >> build fail from exhausted PermGen, so we probably need to increase
> >> MaxPermGen in spark/pom.xml; 2) Not all of the modules got picked up in
> the
> >> recent invocations of the maven-release-plugin -- did your prepare &
> >> release use -Phadoop2-yar,repl-bin , Patrick?
> >>
> >>
> >>
> >> On Wed, Sep 11, 2013 at 10:31 PM, Mark Hamstra <mark@clearstorydata.com
> >wrote:
> >>
> >>> Yes, exactly. If I comment out the reference to the parent Apache pom,
> >>> then the buildnumber plugin works correctly. Similarly if I leave the
> >>> parent Apache pom reference in place but go into my local .m2 cache and
> >>> comment out just the <scm> in the org.apache:apache pom. In other
> words,
> >>> if both <scm>s are present, the buildnumber plugin only sees the SVN
> one
> >>> from the Apache pom, which makes things go wrong.
> >>>
> >>>
> >>> On Wed, Sep 11, 2013 at 9:18 PM, Henry Saputra <
> henry.saputra@gmail.com>wrote:
> >>>
> >>>> Hi Mark,
> >>>>
> >>>> But the spark main pom.xml contains <scm> element that should override
> >>>> <scm> from the parent pom.xml.
> >>>>
> >>>> Are you saying the pom.xml from parent Apache pom.xml overrides the
> >>>> Spark <scm> definition instead?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Sep 11, 2013 at 3:04 PM, Mark Hamstra <
> mark@clearstorydata.com>
> >>>> wrote:
> >>>>> I'm in the process of trying to build Apache-fied Spark into our
> stack
> >>>> at
> >>>>> ClearStory. We've been using Maven to do Debian packaging as found
> in
> >>>> the
> >>>>> repl-bin module for quite a while, but that doesn't work now. To see
> >>>> the
> >>>>> failure, you need to build repl-bin with the `deb` profile -- e.g.
> "mvn
> >>>>> -Prepl-bin,deb install" from the project directory. The problem is
> with
> >>>>> the buildnumber-maven-plugin, which is used to attach the 8-digit
> short
> >>>> SHA
> >>>>> of the current git commit to the Debian package -- so, e.g.,
> >>>>> spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber pl
Re: Maven SCM trouble
Posted by Patrick Wendell <pw...@gmail.com>.
Ya I amended this for pushing the releases to maven. I actually
changed that later to a different URI because it was incorrect (should
have been https://)
On Wed, Sep 11, 2013 at 11:41 PM, Gary Struthers
<ag...@earthlink.net> wrote:
> It looks like scm in the pom is only used to upload artifacts to Maven Central and isn't needed for building. Explained here:
> http://books.sonatype.com/nexus-book/reference/_introduction_7.html
>
> If you run into a transitive dependency problem, there is a nice way to exclude them. Explained here:
> http://books.sonatype.com/mvnref-book/reference/pom-relationships-sect-project-dependencies.html#pom-relationships-sect-conflict
>
> By far the best Maven documentation is in Sonatype's free books. They're accurate and up to date, unlike the Apache Maven docs or StackOverflow.
> http://www.sonatype.com/resources/books
>
> Gary Struthers
>
> On Sep 11, 2013, at 11:21 PM, Mark Hamstra <ma...@clearstorydata.com> wrote:
>
>> Ah hah! Patrick figured it out:
>> https://github.com/apache/incubator-spark/commit/905edf59db662868f55525118131cf102d366587.
>> The buildnumber plugin and Debian packaging works correctly again.
>> Thanks! I'll go through the poms tomorrow and see if there are any other
>> such incompletely overriden elements that could be causing us troubles.
>>
>> Right now I am seeing two problems with the maven build: 1) I just had a
>> build fail from exhausted PermGen, so we probably need to increase
>> MaxPermGen in spark/pom.xml; 2) Not all of the modules got picked up in the
>> recent invocations of the maven-release-plugin -- did your prepare &
>> release use -Phadoop2-yar,repl-bin , Patrick?
>>
>>
>>
>> On Wed, Sep 11, 2013 at 10:31 PM, Mark Hamstra <ma...@clearstorydata.com>wrote:
>>
>>> Yes, exactly. If I comment out the reference to the parent Apache pom,
>>> then the buildnumber plugin works correctly. Similarly if I leave the
>>> parent Apache pom reference in place but go into my local .m2 cache and
>>> comment out just the <scm> in the org.apache:apache pom. In other words,
>>> if both <scm>s are present, the buildnumber plugin only sees the SVN one
>>> from the Apache pom, which makes things go wrong.
>>>
>>>
>>> On Wed, Sep 11, 2013 at 9:18 PM, Henry Saputra <he...@gmail.com>wrote:
>>>
>>>> Hi Mark,
>>>>
>>>> But the spark main pom.xml contains <scm> element that should override
>>>> <scm> from the parent pom.xml.
>>>>
>>>> Are you saying the pom.xml from parent Apache pom.xml overrides the
>>>> Spark <scm> definition instead?
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Sep 11, 2013 at 3:04 PM, Mark Hamstra <ma...@clearstorydata.com>
>>>> wrote:
>>>>> I'm in the process of trying to build Apache-fied Spark into our stack
>>>> at
>>>>> ClearStory. We've been using Maven to do Debian packaging as found in
>>>> the
>>>>> repl-bin module for quite a while, but that doesn't work now. To see
>>>> the
>>>>> failure, you need to build repl-bin with the `deb` profile -- e.g. "mvn
>>>>> -Prepl-bin,deb install" from the project directory. The problem is with
>>>>> the buildnumber-maven-plugin, which is used to attach the 8-digit short
>>>> SHA
>>>>> of the current git commit to the Debian package -- so, e.g.,
>>>>> spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber plugin relies
>>>> upon
>>>>> the <scm> tag to determine which source code management system should be
>>>>> used to lookup the version number.
>>>>>
>>>>> That all worked great until the following was added to spark/pom.xml:
>>>>>
>>>>> <parent>
>>>>> <groupId>org.apache</groupId>
>>>>> <artifactId>apache</artifactId>
>>>>> <version>11</version>
>>>>> </parent>
>>>>>
>>>>> The problem is that the parent pom that the above includes has this
>>>> within
>>>>> it:
>>>>>
>>>>> <scm>
>>>>> <connection>scm:svn:
>>>>> http://svn.apache.org/repos/asf/maven/pom/tags/apache-11</connection
>>>>> <developerConnection>scm:svn:
>>>>> https://svn.apache.org/repos/asf/maven/pom/tags/apache-11
>>>>> </developerConnection>
>>>>> <url>http://svn.apache.org/viewvc/maven/pom/tags/apache-11</url>
>>>>> </scm>
>>>>>
>>>>> And that essentially overrides the <scm> section of spark/pom.xml, which
>>>>> means that the buildnumber plugin thinks that we are using SVN instead
>>>> of
>>>>> git, and the SHA lookup fails.
>>>>>
>>>>> So, that is one known problem in the Maven build. I'm not at all
>>>> certain
>>>>> that there aren't others or lurking problems because of different
>>>> versions
>>>>> of dependencies, plugins, etc. being included by that org.apache:apache
>>>>> artifact and also specified within the Spark pom files.
>>>>>
>>>>> I'm pretty sure that we can't not include the equivalent of
>>>>> org.apache:apache:11, since there is a lot of important stuff in there
>>>> to
>>>>> do Apache release management, etc. So, is there another, equivalent
>>>>> artifact that we can use that doesn't confuse the <scm> issue, or some
>>>>> other way around this problem? It looks like there are some other
>>>>> git-specific plugins out there that we may be able to use instead of
>>>>> buildnumber-maven-plugin, so that may be a valid and necessary solution
>>>> to
>>>>> the Debian packaging problem.
>>>>>
>>>>> Can someone who knows Maven better than I take a look at the overlap
>>>>> between org.apache:apache and the Spark pom files to see if there are
>>>>> problems lurking there? Even if there aren't problems other than with
>>>> the
>>>>> buildnumber plugin, we may be able to trim a lot out of the Spark poms
>>>> that
>>>>> is already present in the apache parent pom.
>>>>
>>>
>>>
>
Re: Maven SCM trouble
Posted by Gary Struthers <ag...@earthlink.net>.
It looks like scm in the pom is only used to upload artifacts to Maven Central and isn't needed for building. Explained here:
http://books.sonatype.com/nexus-book/reference/_introduction_7.html
If you run into a transitive dependency problem, there is a nice way to exclude them. Explained here:
http://books.sonatype.com/mvnref-book/reference/pom-relationships-sect-project-dependencies.html#pom-relationships-sect-conflict
By far the best Maven documentation is in Sonatype's free books. They're accurate and up to date, unlike the Apache Maven docs or StackOverflow.
http://www.sonatype.com/resources/books
Gary Struthers
On Sep 11, 2013, at 11:21 PM, Mark Hamstra <ma...@clearstorydata.com> wrote:
> Ah hah! Patrick figured it out:
> https://github.com/apache/incubator-spark/commit/905edf59db662868f55525118131cf102d366587.
> The buildnumber plugin and Debian packaging works correctly again.
> Thanks! I'll go through the poms tomorrow and see if there are any other
> such incompletely overriden elements that could be causing us troubles.
>
> Right now I am seeing two problems with the maven build: 1) I just had a
> build fail from exhausted PermGen, so we probably need to increase
> MaxPermGen in spark/pom.xml; 2) Not all of the modules got picked up in the
> recent invocations of the maven-release-plugin -- did your prepare &
> release use -Phadoop2-yar,repl-bin , Patrick?
>
>
>
> On Wed, Sep 11, 2013 at 10:31 PM, Mark Hamstra <ma...@clearstorydata.com>wrote:
>
>> Yes, exactly. If I comment out the reference to the parent Apache pom,
>> then the buildnumber plugin works correctly. Similarly if I leave the
>> parent Apache pom reference in place but go into my local .m2 cache and
>> comment out just the <scm> in the org.apache:apache pom. In other words,
>> if both <scm>s are present, the buildnumber plugin only sees the SVN one
>> from the Apache pom, which makes things go wrong.
>>
>>
>> On Wed, Sep 11, 2013 at 9:18 PM, Henry Saputra <he...@gmail.com>wrote:
>>
>>> Hi Mark,
>>>
>>> But the spark main pom.xml contains <scm> element that should override
>>> <scm> from the parent pom.xml.
>>>
>>> Are you saying the pom.xml from parent Apache pom.xml overrides the
>>> Spark <scm> definition instead?
>>>
>>>
>>>
>>>
>>> On Wed, Sep 11, 2013 at 3:04 PM, Mark Hamstra <ma...@clearstorydata.com>
>>> wrote:
>>>> I'm in the process of trying to build Apache-fied Spark into our stack
>>> at
>>>> ClearStory. We've been using Maven to do Debian packaging as found in
>>> the
>>>> repl-bin module for quite a while, but that doesn't work now. To see
>>> the
>>>> failure, you need to build repl-bin with the `deb` profile -- e.g. "mvn
>>>> -Prepl-bin,deb install" from the project directory. The problem is with
>>>> the buildnumber-maven-plugin, which is used to attach the 8-digit short
>>> SHA
>>>> of the current git commit to the Debian package -- so, e.g.,
>>>> spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber plugin relies
>>> upon
>>>> the <scm> tag to determine which source code management system should be
>>>> used to lookup the version number.
>>>>
>>>> That all worked great until the following was added to spark/pom.xml:
>>>>
>>>> <parent>
>>>> <groupId>org.apache</groupId>
>>>> <artifactId>apache</artifactId>
>>>> <version>11</version>
>>>> </parent>
>>>>
>>>> The problem is that the parent pom that the above includes has this
>>> within
>>>> it:
>>>>
>>>> <scm>
>>>> <connection>scm:svn:
>>>> http://svn.apache.org/repos/asf/maven/pom/tags/apache-11</connection
>>>> <developerConnection>scm:svn:
>>>> https://svn.apache.org/repos/asf/maven/pom/tags/apache-11
>>>> </developerConnection>
>>>> <url>http://svn.apache.org/viewvc/maven/pom/tags/apache-11</url>
>>>> </scm>
>>>>
>>>> And that essentially overrides the <scm> section of spark/pom.xml, which
>>>> means that the buildnumber plugin thinks that we are using SVN instead
>>> of
>>>> git, and the SHA lookup fails.
>>>>
>>>> So, that is one known problem in the Maven build. I'm not at all
>>> certain
>>>> that there aren't others or lurking problems because of different
>>> versions
>>>> of dependencies, plugins, etc. being included by that org.apache:apache
>>>> artifact and also specified within the Spark pom files.
>>>>
>>>> I'm pretty sure that we can't not include the equivalent of
>>>> org.apache:apache:11, since there is a lot of important stuff in there
>>> to
>>>> do Apache release management, etc. So, is there another, equivalent
>>>> artifact that we can use that doesn't confuse the <scm> issue, or some
>>>> other way around this problem? It looks like there are some other
>>>> git-specific plugins out there that we may be able to use instead of
>>>> buildnumber-maven-plugin, so that may be a valid and necessary solution
>>> to
>>>> the Debian packaging problem.
>>>>
>>>> Can someone who knows Maven better than I take a look at the overlap
>>>> between org.apache:apache and the Spark pom files to see if there are
>>>> problems lurking there? Even if there aren't problems other than with
>>> the
>>>> buildnumber plugin, we may be able to trim a lot out of the Spark poms
>>> that
>>>> is already present in the apache parent pom.
>>>
>>
>>
Re: Maven SCM trouble
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Ah hah! Patrick figured it out:
https://github.com/apache/incubator-spark/commit/905edf59db662868f55525118131cf102d366587.
The buildnumber plugin and Debian packaging works correctly again.
Thanks! I'll go through the poms tomorrow and see if there are any other
such incompletely overriden elements that could be causing us troubles.
Right now I am seeing two problems with the maven build: 1) I just had a
build fail from exhausted PermGen, so we probably need to increase
MaxPermGen in spark/pom.xml; 2) Not all of the modules got picked up in the
recent invocations of the maven-release-plugin -- did your prepare &
release use -Phadoop2-yar,repl-bin , Patrick?
On Wed, Sep 11, 2013 at 10:31 PM, Mark Hamstra <ma...@clearstorydata.com>wrote:
> Yes, exactly. If I comment out the reference to the parent Apache pom,
> then the buildnumber plugin works correctly. Similarly if I leave the
> parent Apache pom reference in place but go into my local .m2 cache and
> comment out just the <scm> in the org.apache:apache pom. In other words,
> if both <scm>s are present, the buildnumber plugin only sees the SVN one
> from the Apache pom, which makes things go wrong.
>
>
> On Wed, Sep 11, 2013 at 9:18 PM, Henry Saputra <he...@gmail.com>wrote:
>
>> Hi Mark,
>>
>> But the spark main pom.xml contains <scm> element that should override
>> <scm> from the parent pom.xml.
>>
>> Are you saying the pom.xml from parent Apache pom.xml overrides the
>> Spark <scm> definition instead?
>>
>>
>>
>>
>> On Wed, Sep 11, 2013 at 3:04 PM, Mark Hamstra <ma...@clearstorydata.com>
>> wrote:
>> > I'm in the process of trying to build Apache-fied Spark into our stack
>> at
>> > ClearStory. We've been using Maven to do Debian packaging as found in
>> the
>> > repl-bin module for quite a while, but that doesn't work now. To see
>> the
>> > failure, you need to build repl-bin with the `deb` profile -- e.g. "mvn
>> > -Prepl-bin,deb install" from the project directory. The problem is with
>> > the buildnumber-maven-plugin, which is used to attach the 8-digit short
>> SHA
>> > of the current git commit to the Debian package -- so, e.g.,
>> > spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber plugin relies
>> upon
>> > the <scm> tag to determine which source code management system should be
>> > used to lookup the version number.
>> >
>> > That all worked great until the following was added to spark/pom.xml:
>> >
>> > <parent>
>> > <groupId>org.apache</groupId>
>> > <artifactId>apache</artifactId>
>> > <version>11</version>
>> > </parent>
>> >
>> > The problem is that the parent pom that the above includes has this
>> within
>> > it:
>> >
>> > <scm>
>> > <connection>scm:svn:
>> > http://svn.apache.org/repos/asf/maven/pom/tags/apache-11</connection
>> > <developerConnection>scm:svn:
>> > https://svn.apache.org/repos/asf/maven/pom/tags/apache-11
>> > </developerConnection>
>> > <url>http://svn.apache.org/viewvc/maven/pom/tags/apache-11</url>
>> > </scm>
>> >
>> > And that essentially overrides the <scm> section of spark/pom.xml, which
>> > means that the buildnumber plugin thinks that we are using SVN instead
>> of
>> > git, and the SHA lookup fails.
>> >
>> > So, that is one known problem in the Maven build. I'm not at all
>> certain
>> > that there aren't others or lurking problems because of different
>> versions
>> > of dependencies, plugins, etc. being included by that org.apache:apache
>> > artifact and also specified within the Spark pom files.
>> >
>> > I'm pretty sure that we can't not include the equivalent of
>> > org.apache:apache:11, since there is a lot of important stuff in there
>> to
>> > do Apache release management, etc. So, is there another, equivalent
>> > artifact that we can use that doesn't confuse the <scm> issue, or some
>> > other way around this problem? It looks like there are some other
>> > git-specific plugins out there that we may be able to use instead of
>> > buildnumber-maven-plugin, so that may be a valid and necessary solution
>> to
>> > the Debian packaging problem.
>> >
>> > Can someone who knows Maven better than I take a look at the overlap
>> > between org.apache:apache and the Spark pom files to see if there are
>> > problems lurking there? Even if there aren't problems other than with
>> the
>> > buildnumber plugin, we may be able to trim a lot out of the Spark poms
>> that
>> > is already present in the apache parent pom.
>>
>
>
Re: Maven SCM trouble
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Yes, exactly. If I comment out the reference to the parent Apache pom,
then the buildnumber plugin works correctly. Similarly if I leave the
parent Apache pom reference in place but go into my local .m2 cache and
comment out just the <scm> in the org.apache:apache pom. In other words,
if both <scm>s are present, the buildnumber plugin only sees the SVN one
from the Apache pom, which makes things go wrong.
On Wed, Sep 11, 2013 at 9:18 PM, Henry Saputra <he...@gmail.com>wrote:
> Hi Mark,
>
> But the spark main pom.xml contains <scm> element that should override
> <scm> from the parent pom.xml.
>
> Are you saying the pom.xml from parent Apache pom.xml overrides the
> Spark <scm> definition instead?
>
>
>
>
> On Wed, Sep 11, 2013 at 3:04 PM, Mark Hamstra <ma...@clearstorydata.com>
> wrote:
> > I'm in the process of trying to build Apache-fied Spark into our stack at
> > ClearStory. We've been using Maven to do Debian packaging as found in
> the
> > repl-bin module for quite a while, but that doesn't work now. To see the
> > failure, you need to build repl-bin with the `deb` profile -- e.g. "mvn
> > -Prepl-bin,deb install" from the project directory. The problem is with
> > the buildnumber-maven-plugin, which is used to attach the 8-digit short
> SHA
> > of the current git commit to the Debian package -- so, e.g.,
> > spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber plugin relies
> upon
> > the <scm> tag to determine which source code management system should be
> > used to lookup the version number.
> >
> > That all worked great until the following was added to spark/pom.xml:
> >
> > <parent>
> > <groupId>org.apache</groupId>
> > <artifactId>apache</artifactId>
> > <version>11</version>
> > </parent>
> >
> > The problem is that the parent pom that the above includes has this
> within
> > it:
> >
> > <scm>
> > <connection>scm:svn:
> > http://svn.apache.org/repos/asf/maven/pom/tags/apache-11</connection
> > <developerConnection>scm:svn:
> > https://svn.apache.org/repos/asf/maven/pom/tags/apache-11
> > </developerConnection>
> > <url>http://svn.apache.org/viewvc/maven/pom/tags/apache-11</url>
> > </scm>
> >
> > And that essentially overrides the <scm> section of spark/pom.xml, which
> > means that the buildnumber plugin thinks that we are using SVN instead of
> > git, and the SHA lookup fails.
> >
> > So, that is one known problem in the Maven build. I'm not at all certain
> > that there aren't others or lurking problems because of different
> versions
> > of dependencies, plugins, etc. being included by that org.apache:apache
> > artifact and also specified within the Spark pom files.
> >
> > I'm pretty sure that we can't not include the equivalent of
> > org.apache:apache:11, since there is a lot of important stuff in there to
> > do Apache release management, etc. So, is there another, equivalent
> > artifact that we can use that doesn't confuse the <scm> issue, or some
> > other way around this problem? It looks like there are some other
> > git-specific plugins out there that we may be able to use instead of
> > buildnumber-maven-plugin, so that may be a valid and necessary solution
> to
> > the Debian packaging problem.
> >
> > Can someone who knows Maven better than I take a look at the overlap
> > between org.apache:apache and the Spark pom files to see if there are
> > problems lurking there? Even if there aren't problems other than with
> the
> > buildnumber plugin, we may be able to trim a lot out of the Spark poms
> that
> > is already present in the apache parent pom.
>
Re: Maven SCM trouble
Posted by Henry Saputra <he...@gmail.com>.
Hi Mark,
But the spark main pom.xml contains <scm> element that should override
<scm> from the parent pom.xml.
Are you saying the pom.xml from parent Apache pom.xml overrides the
Spark <scm> definition instead?
On Wed, Sep 11, 2013 at 3:04 PM, Mark Hamstra <ma...@clearstorydata.com> wrote:
> I'm in the process of trying to build Apache-fied Spark into our stack at
> ClearStory. We've been using Maven to do Debian packaging as found in the
> repl-bin module for quite a while, but that doesn't work now. To see the
> failure, you need to build repl-bin with the `deb` profile -- e.g. "mvn
> -Prepl-bin,deb install" from the project directory. The problem is with
> the buildnumber-maven-plugin, which is used to attach the 8-digit short SHA
> of the current git commit to the Debian package -- so, e.g.,
> spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber plugin relies upon
> the <scm> tag to determine which source code management system should be
> used to lookup the version number.
>
> That all worked great until the following was added to spark/pom.xml:
>
> <parent>
> <groupId>org.apache</groupId>
> <artifactId>apache</artifactId>
> <version>11</version>
> </parent>
>
> The problem is that the parent pom that the above includes has this within
> it:
>
> <scm>
> <connection>scm:svn:
> http://svn.apache.org/repos/asf/maven/pom/tags/apache-11</connection
> <developerConnection>scm:svn:
> https://svn.apache.org/repos/asf/maven/pom/tags/apache-11
> </developerConnection>
> <url>http://svn.apache.org/viewvc/maven/pom/tags/apache-11</url>
> </scm>
>
> And that essentially overrides the <scm> section of spark/pom.xml, which
> means that the buildnumber plugin thinks that we are using SVN instead of
> git, and the SHA lookup fails.
>
> So, that is one known problem in the Maven build. I'm not at all certain
> that there aren't others or lurking problems because of different versions
> of dependencies, plugins, etc. being included by that org.apache:apache
> artifact and also specified within the Spark pom files.
>
> I'm pretty sure that we can't not include the equivalent of
> org.apache:apache:11, since there is a lot of important stuff in there to
> do Apache release management, etc. So, is there another, equivalent
> artifact that we can use that doesn't confuse the <scm> issue, or some
> other way around this problem? It looks like there are some other
> git-specific plugins out there that we may be able to use instead of
> buildnumber-maven-plugin, so that may be a valid and necessary solution to
> the Debian packaging problem.
>
> Can someone who knows Maven better than I take a look at the overlap
> between org.apache:apache and the Spark pom files to see if there are
> problems lurking there? Even if there aren't problems other than with the
> buildnumber plugin, we may be able to trim a lot out of the Spark poms that
> is already present in the apache parent pom.