You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Patrick Wendell <pw...@gmail.com> on 2014/11/14 21:17:54 UTC
Has anyone else observed this build break?
A recent patch broke clean builds for me, I am trying to see how
widespread this issue is and whether we need to revert the patch.
The error I've seen is this when building the examples project:
spark-examples_2.10: Could not resolve dependencies for project
org.apache.spark:spark-examples_2.10:jar:1.2.0-SNAPSHOT: Could not
find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/../lib/tools.jar
The reason for this error is that hbase-annotations is using a
"system" scoped dependency in their hbase-annotations pom, and this
doesn't work with certain JDK layouts such as that provided on Mac OS:
http://central.maven.org/maven2/org/apache/hbase/hbase-annotations/0.98.7-hadoop2/hbase-annotations-0.98.7-hadoop2.pom
Has anyone else seen this or is it just me?
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Has anyone else observed this build break?
Posted by Patrick Wendell <pw...@gmail.com>.
Sounds like this is pretty specific to my environment so not a big
deal then. However, if we can safely exclude those packages it's worth
doing.
On Sat, Nov 15, 2014 at 7:27 AM, Ted Yu <yu...@gmail.com> wrote:
> I couldn't reproduce the problem using:
>
> java version "1.6.0_65"
> Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
> Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
>
> Since hbase-annotations is a transitive dependency, I created the following
> pull request to exclude it from various hbase modules:
> https://github.com/apache/spark/pull/3286
>
> Cheers
>
> https://github.com/apache/spark/pull/3286
>
> On Sat, Nov 15, 2014 at 6:56 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> Sorry for the late reply.
>>
>> I tested my patch on Mac with the following JDK:
>>
>> java version "1.7.0_60"
>> Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
>> Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)
>>
>> Let me see if the problem can be solved upstream in HBase
>> hbase-annotations module.
>>
>> Cheers
>>
>> On Fri, Nov 14, 2014 at 12:32 PM, Patrick Wendell <pw...@gmail.com>
>> wrote:
>>>
>>> I think in this case we can probably just drop that dependency, so
>>> there is a simpler fix. But mostly I'm curious whether anyone else has
>>> observed this.
>>>
>>> On Fri, Nov 14, 2014 at 12:24 PM, Hari Shreedharan
>>> <hs...@cloudera.com> wrote:
>>> > Seems like a comment on that page mentions a fix, which would add yet
>>> > another profile though -- specifically telling mvn that if it is an
>>> > apple
>>> > jdk, use the classes.jar as the tools.jar as well, since Apple-packaged
>>> > JDK
>>> > 6 bundled them together.
>>> >
>>> > Link:
>>> > http://permalink.gmane.org/gmane.comp.java.maven-plugins.mojo.user/4320
>>> >
>>> > I didn't test it, but maybe this can fix it?
>>> >
>>> > Thanks,
>>> > Hari
>>> >
>>> >
>>> > On Fri, Nov 14, 2014 at 12:21 PM, Patrick Wendell <pw...@gmail.com>
>>> > wrote:
>>> >>
>>> >> A work around for this fix is identified here:
>>> >>
>>> >>
>>> >> http://dbknickerbocker.blogspot.com/2013/04/simple-fix-to-missing-toolsjar-in-jdk.html
>>> >>
>>> >> However, if this affects more users I'd prefer to just fix it properly
>>> >> in our build.
>>> >>
>>> >> On Fri, Nov 14, 2014 at 12:17 PM, Patrick Wendell <pw...@gmail.com>
>>> >> wrote:
>>> >> > A recent patch broke clean builds for me, I am trying to see how
>>> >> > widespread this issue is and whether we need to revert the patch.
>>> >> >
>>> >> > The error I've seen is this when building the examples project:
>>> >> >
>>> >> > spark-examples_2.10: Could not resolve dependencies for project
>>> >> > org.apache.spark:spark-examples_2.10:jar:1.2.0-SNAPSHOT: Could not
>>> >> > find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
>>> >> >
>>> >> >
>>> >> > /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/../lib/tools.jar
>>> >> >
>>> >> > The reason for this error is that hbase-annotations is using a
>>> >> > "system" scoped dependency in their hbase-annotations pom, and this
>>> >> > doesn't work with certain JDK layouts such as that provided on Mac
>>> >> > OS:
>>> >> >
>>> >> >
>>> >> >
>>> >> > http://central.maven.org/maven2/org/apache/hbase/hbase-annotations/0.98.7-hadoop2/hbase-annotations-0.98.7-hadoop2.pom
>>> >> >
>>> >> > Has anyone else seen this or is it just me?
>>> >> >
>>> >> > - Patrick
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> >> For additional commands, e-mail: dev-help@spark.apache.org
>>> >>
>>> >
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Has anyone else observed this build break?
Posted by Ted Yu <yu...@gmail.com>.
I couldn't reproduce the problem using:
java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
Since hbase-annotations is a transitive dependency, I created the following
pull request to exclude it from various hbase modules:
https://github.com/apache/spark/pull/3286
Cheers
https://github.com/apache/spark/pull/3286
On Sat, Nov 15, 2014 at 6:56 AM, Ted Yu <yu...@gmail.com> wrote:
> Sorry for the late reply.
>
> I tested my patch on Mac with the following JDK:
>
> java version "1.7.0_60"
> Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
> Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)
>
> Let me see if the problem can be solved upstream in HBase hbase-annotations
> module.
>
> Cheers
>
> On Fri, Nov 14, 2014 at 12:32 PM, Patrick Wendell <pw...@gmail.com>
> wrote:
>
>> I think in this case we can probably just drop that dependency, so
>> there is a simpler fix. But mostly I'm curious whether anyone else has
>> observed this.
>>
>> On Fri, Nov 14, 2014 at 12:24 PM, Hari Shreedharan
>> <hs...@cloudera.com> wrote:
>> > Seems like a comment on that page mentions a fix, which would add yet
>> > another profile though -- specifically telling mvn that if it is an
>> apple
>> > jdk, use the classes.jar as the tools.jar as well, since Apple-packaged
>> JDK
>> > 6 bundled them together.
>> >
>> > Link:
>> > http://permalink.gmane.org/gmane.comp.java.maven-plugins.mojo.user/4320
>> >
>> > I didn't test it, but maybe this can fix it?
>> >
>> > Thanks,
>> > Hari
>> >
>> >
>> > On Fri, Nov 14, 2014 at 12:21 PM, Patrick Wendell <pw...@gmail.com>
>> > wrote:
>> >>
>> >> A work around for this fix is identified here:
>> >>
>> >>
>> http://dbknickerbocker.blogspot.com/2013/04/simple-fix-to-missing-toolsjar-in-jdk.html
>> >>
>> >> However, if this affects more users I'd prefer to just fix it properly
>> >> in our build.
>> >>
>> >> On Fri, Nov 14, 2014 at 12:17 PM, Patrick Wendell <pw...@gmail.com>
>> >> wrote:
>> >> > A recent patch broke clean builds for me, I am trying to see how
>> >> > widespread this issue is and whether we need to revert the patch.
>> >> >
>> >> > The error I've seen is this when building the examples project:
>> >> >
>> >> > spark-examples_2.10: Could not resolve dependencies for project
>> >> > org.apache.spark:spark-examples_2.10:jar:1.2.0-SNAPSHOT: Could not
>> >> > find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
>> >> >
>> >> >
>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/../lib/tools.jar
>> >> >
>> >> > The reason for this error is that hbase-annotations is using a
>> >> > "system" scoped dependency in their hbase-annotations pom, and this
>> >> > doesn't work with certain JDK layouts such as that provided on Mac
>> OS:
>> >> >
>> >> >
>> >> >
>> http://central.maven.org/maven2/org/apache/hbase/hbase-annotations/0.98.7-hadoop2/hbase-annotations-0.98.7-hadoop2.pom
>> >> >
>> >> > Has anyone else seen this or is it just me?
>> >> >
>> >> > - Patrick
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >> For additional commands, e-mail: dev-help@spark.apache.org
>> >>
>> >
>>
>
>
Re: Has anyone else observed this build break?
Posted by Ted Yu <yu...@gmail.com>.
Sorry for the late reply.
I tested my patch on Mac with the following JDK:
java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)
Let me see if the problem can be solved upstream in HBase hbase-annotations
module.
Cheers
On Fri, Nov 14, 2014 at 12:32 PM, Patrick Wendell <pw...@gmail.com>
wrote:
> I think in this case we can probably just drop that dependency, so
> there is a simpler fix. But mostly I'm curious whether anyone else has
> observed this.
>
> On Fri, Nov 14, 2014 at 12:24 PM, Hari Shreedharan
> <hs...@cloudera.com> wrote:
> > Seems like a comment on that page mentions a fix, which would add yet
> > another profile though -- specifically telling mvn that if it is an apple
> > jdk, use the classes.jar as the tools.jar as well, since Apple-packaged
> JDK
> > 6 bundled them together.
> >
> > Link:
> > http://permalink.gmane.org/gmane.comp.java.maven-plugins.mojo.user/4320
> >
> > I didn't test it, but maybe this can fix it?
> >
> > Thanks,
> > Hari
> >
> >
> > On Fri, Nov 14, 2014 at 12:21 PM, Patrick Wendell <pw...@gmail.com>
> > wrote:
> >>
> >> A work around for this fix is identified here:
> >>
> >>
> http://dbknickerbocker.blogspot.com/2013/04/simple-fix-to-missing-toolsjar-in-jdk.html
> >>
> >> However, if this affects more users I'd prefer to just fix it properly
> >> in our build.
> >>
> >> On Fri, Nov 14, 2014 at 12:17 PM, Patrick Wendell <pw...@gmail.com>
> >> wrote:
> >> > A recent patch broke clean builds for me, I am trying to see how
> >> > widespread this issue is and whether we need to revert the patch.
> >> >
> >> > The error I've seen is this when building the examples project:
> >> >
> >> > spark-examples_2.10: Could not resolve dependencies for project
> >> > org.apache.spark:spark-examples_2.10:jar:1.2.0-SNAPSHOT: Could not
> >> > find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
> >> >
> >> >
> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/../lib/tools.jar
> >> >
> >> > The reason for this error is that hbase-annotations is using a
> >> > "system" scoped dependency in their hbase-annotations pom, and this
> >> > doesn't work with certain JDK layouts such as that provided on Mac OS:
> >> >
> >> >
> >> >
> http://central.maven.org/maven2/org/apache/hbase/hbase-annotations/0.98.7-hadoop2/hbase-annotations-0.98.7-hadoop2.pom
> >> >
> >> > Has anyone else seen this or is it just me?
> >> >
> >> > - Patrick
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: dev-help@spark.apache.org
> >>
> >
>
Re: Has anyone else observed this build break?
Posted by Patrick Wendell <pw...@gmail.com>.
I think in this case we can probably just drop that dependency, so
there is a simpler fix. But mostly I'm curious whether anyone else has
observed this.
On Fri, Nov 14, 2014 at 12:24 PM, Hari Shreedharan
<hs...@cloudera.com> wrote:
> Seems like a comment on that page mentions a fix, which would add yet
> another profile though -- specifically telling mvn that if it is an apple
> jdk, use the classes.jar as the tools.jar as well, since Apple-packaged JDK
> 6 bundled them together.
>
> Link:
> http://permalink.gmane.org/gmane.comp.java.maven-plugins.mojo.user/4320
>
> I didn't test it, but maybe this can fix it?
>
> Thanks,
> Hari
>
>
> On Fri, Nov 14, 2014 at 12:21 PM, Patrick Wendell <pw...@gmail.com>
> wrote:
>>
>> A work around for this fix is identified here:
>>
>> http://dbknickerbocker.blogspot.com/2013/04/simple-fix-to-missing-toolsjar-in-jdk.html
>>
>> However, if this affects more users I'd prefer to just fix it properly
>> in our build.
>>
>> On Fri, Nov 14, 2014 at 12:17 PM, Patrick Wendell <pw...@gmail.com>
>> wrote:
>> > A recent patch broke clean builds for me, I am trying to see how
>> > widespread this issue is and whether we need to revert the patch.
>> >
>> > The error I've seen is this when building the examples project:
>> >
>> > spark-examples_2.10: Could not resolve dependencies for project
>> > org.apache.spark:spark-examples_2.10:jar:1.2.0-SNAPSHOT: Could not
>> > find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
>> >
>> > /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/../lib/tools.jar
>> >
>> > The reason for this error is that hbase-annotations is using a
>> > "system" scoped dependency in their hbase-annotations pom, and this
>> > doesn't work with certain JDK layouts such as that provided on Mac OS:
>> >
>> >
>> > http://central.maven.org/maven2/org/apache/hbase/hbase-annotations/0.98.7-hadoop2/hbase-annotations-0.98.7-hadoop2.pom
>> >
>> > Has anyone else seen this or is it just me?
>> >
>> > - Patrick
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Has anyone else observed this build break?
Posted by Hari Shreedharan <hs...@cloudera.com>.
Seems like a comment on that page mentions a fix, which would add yet another profile though — specifically telling mvn that if it is an apple jdk, use the classes.jar as the tools.jar as well, since Apple-packaged JDK 6 bundled them together.
Link: http://permalink.gmane.org/gmane.comp.java.maven-plugins.mojo.user/4320
I didn’t test it, but maybe this can fix it?
Thanks,
Hari
On Fri, Nov 14, 2014 at 12:21 PM, Patrick Wendell <pw...@gmail.com>
wrote:
> A work around for this fix is identified here:
> http://dbknickerbocker.blogspot.com/2013/04/simple-fix-to-missing-toolsjar-in-jdk.html
> However, if this affects more users I'd prefer to just fix it properly
> in our build.
> On Fri, Nov 14, 2014 at 12:17 PM, Patrick Wendell <pw...@gmail.com> wrote:
>> A recent patch broke clean builds for me, I am trying to see how
>> widespread this issue is and whether we need to revert the patch.
>>
>> The error I've seen is this when building the examples project:
>>
>> spark-examples_2.10: Could not resolve dependencies for project
>> org.apache.spark:spark-examples_2.10:jar:1.2.0-SNAPSHOT: Could not
>> find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/../lib/tools.jar
>>
>> The reason for this error is that hbase-annotations is using a
>> "system" scoped dependency in their hbase-annotations pom, and this
>> doesn't work with certain JDK layouts such as that provided on Mac OS:
>>
>> http://central.maven.org/maven2/org/apache/hbase/hbase-annotations/0.98.7-hadoop2/hbase-annotations-0.98.7-hadoop2.pom
>>
>> Has anyone else seen this or is it just me?
>>
>> - Patrick
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
Re: Has anyone else observed this build break?
Posted by Patrick Wendell <pw...@gmail.com>.
A work around for this fix is identified here:
http://dbknickerbocker.blogspot.com/2013/04/simple-fix-to-missing-toolsjar-in-jdk.html
However, if this affects more users I'd prefer to just fix it properly
in our build.
On Fri, Nov 14, 2014 at 12:17 PM, Patrick Wendell <pw...@gmail.com> wrote:
> A recent patch broke clean builds for me, I am trying to see how
> widespread this issue is and whether we need to revert the patch.
>
> The error I've seen is this when building the examples project:
>
> spark-examples_2.10: Could not resolve dependencies for project
> org.apache.spark:spark-examples_2.10:jar:1.2.0-SNAPSHOT: Could not
> find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/../lib/tools.jar
>
> The reason for this error is that hbase-annotations is using a
> "system" scoped dependency in their hbase-annotations pom, and this
> doesn't work with certain JDK layouts such as that provided on Mac OS:
>
> http://central.maven.org/maven2/org/apache/hbase/hbase-annotations/0.98.7-hadoop2/hbase-annotations-0.98.7-hadoop2.pom
>
> Has anyone else seen this or is it just me?
>
> - Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Has anyone else observed this build break?
Posted by Sean Owen <so...@cloudera.com>.
FWIW I do not see this on master with "mvn -DskipTests clean package".
I'm on OS X 10.10 and I build with Java 8 by default.
On Fri, Nov 14, 2014 at 8:17 PM, Patrick Wendell <pw...@gmail.com> wrote:
> A recent patch broke clean builds for me, I am trying to see how
> widespread this issue is and whether we need to revert the patch.
>
> The error I've seen is this when building the examples project:
>
> spark-examples_2.10: Could not resolve dependencies for project
> org.apache.spark:spark-examples_2.10:jar:1.2.0-SNAPSHOT: Could not
> find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/../lib/tools.jar
>
> The reason for this error is that hbase-annotations is using a
> "system" scoped dependency in their hbase-annotations pom, and this
> doesn't work with certain JDK layouts such as that provided on Mac OS:
>
> http://central.maven.org/maven2/org/apache/hbase/hbase-annotations/0.98.7-hadoop2/hbase-annotations-0.98.7-hadoop2.pom
>
> Has anyone else seen this or is it just me?
>
> - Patrick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org