You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Walrus theCat <wa...@gmail.com> on 2015/01/18 00:43:40 UTC
maven doesn't build dependencies with Scala 2.11
Hi,
When I run this:
dev/change-version-to-2.11.sh
mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
as per here
<https://spark.apache.org/docs/latest/building-spark.html#building-for-scala-211>,
maven doesn't build Spark's dependencies.
Only when I run:
dev/change-version-to-2.11.sh
sbt/sbt -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
as gathered from here
<https://github.com/ScrapCodes/spark-1/blob/patch-3/docs/building-spark.md>,
do I get Spark's dependencies built without any cross-compilation
errors.
*Question*:
- How can I make maven do this?
- How can I specify the use of Scala 2.11 in my own .pom files?
Thanks
Re: maven doesn't build dependencies with Scala 2.11
Posted by Ted Yu <yu...@gmail.com>.
Now that Kafka 0.8.2.0 has been released, adding external/kafka module
works.
FYI
On Sun, Jan 18, 2015 at 7:36 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. there was no 2.11 Kafka available
>
> That's right. Adding external/kafka module resulted in:
>
> [ERROR] Failed to execute goal on project spark-streaming-kafka_2.11:
> Could not resolve dependencies for project
> org.apache.spark:spark-streaming-kafka_2.11:jar:1.3.0-SNAPSHOT: Could not
> find artifact org.apache.kafka:kafka_2.11:jar:0.8.0 in central (
> https://repo1.maven.org/maven2) -> [Help 1]
>
> Cheers
>
> On Sun, Jan 18, 2015 at 10:41 AM, Sean Owen <so...@cloudera.com> wrote:
>
>> I could be wrong, but I thought this was on purpose. At the time it
>> was set up, there was no 2.11 Kafka available? or one of its
>> dependencies wouldn't work with 2.11?
>>
>> But I'm not sure what the OP means by "maven doesn't build Spark's
>> dependencies" because Ted indicates it does, and of course you can see
>> that these artifacts are published.
>>
>> On Sun, Jan 18, 2015 at 2:46 AM, Ted Yu <yu...@gmail.com> wrote:
>> > There're 3 jars under lib_managed/jars directory with and without
>> > -Dscala-2.11 flag.
>> >
>> > Difference between scala-2.10 and scala-2.11 profiles is that scala-2.10
>> > profile has the following:
>> > <modules>
>> > <module>external/kafka</module>
>> > </modules>
>> >
>> > FYI
>> >
>> > On Sat, Jan 17, 2015 at 4:07 PM, Ted Yu <yu...@gmail.com> wrote:
>> >>
>> >> I did the following:
>> >> 1655 dev/change-version-to-2.11.sh
>> >> 1657 mvn -DHADOOP_PROFILE=hadoop-2.4 -Pyarn,hive -Phadoop-2.4
>> >> -Dscala-2.11 -DskipTests clean package
>> >>
>> >> And mvn command passed.
>> >>
>> >> Did you see any cross-compilation errors ?
>> >>
>> >> Cheers
>> >>
>> >> BTW the two links you mentioned are consistent in terms of building for
>> >> Scala 2.11
>> >>
>> >> On Sat, Jan 17, 2015 at 3:43 PM, Walrus theCat <walrusthecat@gmail.com
>> >
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> When I run this:
>> >>>
>> >>> dev/change-version-to-2.11.sh
>> >>> mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
>> >>>
>> >>> as per here, maven doesn't build Spark's dependencies.
>> >>>
>> >>> Only when I run:
>> >>>
>> >>> dev/change-version-to-2.11.sh
>> >>> sbt/sbt -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
>> >>>
>> >>> as gathered from here, do I get Spark's dependencies built without any
>> >>> cross-compilation errors.
>> >>>
>> >>> Question:
>> >>>
>> >>> - How can I make maven do this?
>> >>>
>> >>> - How can I specify the use of Scala 2.11 in my own .pom files?
>> >>>
>> >>> Thanks
>> >>
>> >>
>> >
>>
>
>
Re: maven doesn't build dependencies with Scala 2.11
Posted by Ted Yu <yu...@gmail.com>.
bq. there was no 2.11 Kafka available
That's right. Adding external/kafka module resulted in:
[ERROR] Failed to execute goal on project spark-streaming-kafka_2.11: Could
not resolve dependencies for project
org.apache.spark:spark-streaming-kafka_2.11:jar:1.3.0-SNAPSHOT: Could not
find artifact org.apache.kafka:kafka_2.11:jar:0.8.0 in central (
https://repo1.maven.org/maven2) -> [Help 1]
Cheers
On Sun, Jan 18, 2015 at 10:41 AM, Sean Owen <so...@cloudera.com> wrote:
> I could be wrong, but I thought this was on purpose. At the time it
> was set up, there was no 2.11 Kafka available? or one of its
> dependencies wouldn't work with 2.11?
>
> But I'm not sure what the OP means by "maven doesn't build Spark's
> dependencies" because Ted indicates it does, and of course you can see
> that these artifacts are published.
>
> On Sun, Jan 18, 2015 at 2:46 AM, Ted Yu <yu...@gmail.com> wrote:
> > There're 3 jars under lib_managed/jars directory with and without
> > -Dscala-2.11 flag.
> >
> > Difference between scala-2.10 and scala-2.11 profiles is that scala-2.10
> > profile has the following:
> > <modules>
> > <module>external/kafka</module>
> > </modules>
> >
> > FYI
> >
> > On Sat, Jan 17, 2015 at 4:07 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> I did the following:
> >> 1655 dev/change-version-to-2.11.sh
> >> 1657 mvn -DHADOOP_PROFILE=hadoop-2.4 -Pyarn,hive -Phadoop-2.4
> >> -Dscala-2.11 -DskipTests clean package
> >>
> >> And mvn command passed.
> >>
> >> Did you see any cross-compilation errors ?
> >>
> >> Cheers
> >>
> >> BTW the two links you mentioned are consistent in terms of building for
> >> Scala 2.11
> >>
> >> On Sat, Jan 17, 2015 at 3:43 PM, Walrus theCat <wa...@gmail.com>
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> When I run this:
> >>>
> >>> dev/change-version-to-2.11.sh
> >>> mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
> >>>
> >>> as per here, maven doesn't build Spark's dependencies.
> >>>
> >>> Only when I run:
> >>>
> >>> dev/change-version-to-2.11.sh
> >>> sbt/sbt -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
> >>>
> >>> as gathered from here, do I get Spark's dependencies built without any
> >>> cross-compilation errors.
> >>>
> >>> Question:
> >>>
> >>> - How can I make maven do this?
> >>>
> >>> - How can I specify the use of Scala 2.11 in my own .pom files?
> >>>
> >>> Thanks
> >>
> >>
> >
>
Re: maven doesn't build dependencies with Scala 2.11
Posted by Sean Owen <so...@cloudera.com>.
I could be wrong, but I thought this was on purpose. At the time it
was set up, there was no 2.11 Kafka available? or one of its
dependencies wouldn't work with 2.11?
But I'm not sure what the OP means by "maven doesn't build Spark's
dependencies" because Ted indicates it does, and of course you can see
that these artifacts are published.
On Sun, Jan 18, 2015 at 2:46 AM, Ted Yu <yu...@gmail.com> wrote:
> There're 3 jars under lib_managed/jars directory with and without
> -Dscala-2.11 flag.
>
> Difference between scala-2.10 and scala-2.11 profiles is that scala-2.10
> profile has the following:
> <modules>
> <module>external/kafka</module>
> </modules>
>
> FYI
>
> On Sat, Jan 17, 2015 at 4:07 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> I did the following:
>> 1655 dev/change-version-to-2.11.sh
>> 1657 mvn -DHADOOP_PROFILE=hadoop-2.4 -Pyarn,hive -Phadoop-2.4
>> -Dscala-2.11 -DskipTests clean package
>>
>> And mvn command passed.
>>
>> Did you see any cross-compilation errors ?
>>
>> Cheers
>>
>> BTW the two links you mentioned are consistent in terms of building for
>> Scala 2.11
>>
>> On Sat, Jan 17, 2015 at 3:43 PM, Walrus theCat <wa...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> When I run this:
>>>
>>> dev/change-version-to-2.11.sh
>>> mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
>>>
>>> as per here, maven doesn't build Spark's dependencies.
>>>
>>> Only when I run:
>>>
>>> dev/change-version-to-2.11.sh
>>> sbt/sbt -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
>>>
>>> as gathered from here, do I get Spark's dependencies built without any
>>> cross-compilation errors.
>>>
>>> Question:
>>>
>>> - How can I make maven do this?
>>>
>>> - How can I specify the use of Scala 2.11 in my own .pom files?
>>>
>>> Thanks
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: maven doesn't build dependencies with Scala 2.11
Posted by Ted Yu <yu...@gmail.com>.
There're 3 jars under lib_managed/jars directory with and
without -Dscala-2.11 flag.
Difference between scala-2.10 and scala-2.11 profiles is that scala-2.10
profile has the following:
<modules>
<module>external/kafka</module>
</modules>
FYI
On Sat, Jan 17, 2015 at 4:07 PM, Ted Yu <yu...@gmail.com> wrote:
> I did the following:
> 1655 dev/change-version-to-2.11.sh
> 1657 mvn -DHADOOP_PROFILE=hadoop-2.4 -Pyarn,hive -Phadoop-2.4
> -Dscala-2.11 -DskipTests clean package
>
> And mvn command passed.
>
> Did you see any cross-compilation errors ?
>
> Cheers
>
> BTW the two links you mentioned are consistent in terms of building for
> Scala 2.11
>
> On Sat, Jan 17, 2015 at 3:43 PM, Walrus theCat <wa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> When I run this:
>>
>> dev/change-version-to-2.11.sh
>> mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
>>
>> as per here
>> <https://spark.apache.org/docs/latest/building-spark.html#building-for-scala-211>,
>> maven doesn't build Spark's dependencies.
>>
>> Only when I run:
>>
>> dev/change-version-to-2.11.sh
>> sbt/sbt -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
>>
>> as gathered from here <https://github.com/ScrapCodes/spark-1/blob/patch-3/docs/building-spark.md>, do I get Spark's dependencies built without any cross-compilation errors.
>>
>> *Question*:
>>
>> - How can I make maven do this?
>>
>> - How can I specify the use of Scala 2.11 in my own .pom files?
>>
>> Thanks
>>
>>
>
Re: maven doesn't build dependencies with Scala 2.11
Posted by Ted Yu <yu...@gmail.com>.
I did the following:
1655 dev/change-version-to-2.11.sh
1657 mvn -DHADOOP_PROFILE=hadoop-2.4 -Pyarn,hive -Phadoop-2.4
-Dscala-2.11 -DskipTests clean package
And mvn command passed.
Did you see any cross-compilation errors ?
Cheers
BTW the two links you mentioned are consistent in terms of building for
Scala 2.11
On Sat, Jan 17, 2015 at 3:43 PM, Walrus theCat <wa...@gmail.com>
wrote:
> Hi,
>
> When I run this:
>
> dev/change-version-to-2.11.sh
> mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
>
> as per here
> <https://spark.apache.org/docs/latest/building-spark.html#building-for-scala-211>,
> maven doesn't build Spark's dependencies.
>
> Only when I run:
>
> dev/change-version-to-2.11.sh
> sbt/sbt -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
>
> as gathered from here <https://github.com/ScrapCodes/spark-1/blob/patch-3/docs/building-spark.md>, do I get Spark's dependencies built without any cross-compilation errors.
>
> *Question*:
>
> - How can I make maven do this?
>
> - How can I specify the use of Scala 2.11 in my own .pom files?
>
> Thanks
>
>