You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2016/09/17 19:14:11 UTC
Different versions of dependencies in assembly/target/scala-2.11/jars?
Hi,
Just noticed in assembly/target/scala-2.11/jars that similar libraries
have different versions:
-rw-r--r-- 1 jacek staff 1230201 17 wrz 09:51 netty-3.8.0.Final.jar
-rw-r--r-- 1 jacek staff 2305335 17 wrz 09:51 netty-all-4.0.41.Final.jar
and
-rw-r--r-- 1 jacek staff 218076 17 wrz 09:51 parquet-hadoop-1.8.1.jar
-rw-r--r-- 1 jacek staff 2796935 17 wrz 09:51
parquet-hadoop-bundle-1.6.0.jar
and
-rw-r--r-- 1 jacek staff 46983 17 wrz 09:51 jackson-annotations-2.6.5.jar
-rw-r--r-- 1 jacek staff 258876 17 wrz 09:51 jackson-core-2.6.5.jar
-rw-r--r-- 1 jacek staff 232248 17 wrz 09:51 jackson-core-asl-1.9.13.jar
-rw-r--r-- 1 jacek staff 1171380 17 wrz 09:51 jackson-databind-2.6.5.jar
-rw-r--r-- 1 jacek staff 18336 17 wrz 09:51 jackson-jaxrs-1.9.13.jar
-rw-r--r-- 1 jacek staff 780664 17 wrz 09:51 jackson-mapper-asl-1.9.13.jar
-rw-r--r-- 1 jacek staff 41263 17 wrz 09:51
jackson-module-paranamer-2.6.5.jar
-rw-r--r-- 1 jacek staff 515604 17 wrz 09:51
jackson-module-scala_2.11-2.6.5.jar
-rw-r--r-- 1 jacek staff 27084 17 wrz 09:51 jackson-xc-1.9.13.jar
and
-rw-r--r-- 1 jacek staff 188671 17 wrz 09:51 commons-beanutils-1.7.0.jar
-rw-r--r-- 1 jacek staff 206035 17 wrz 09:51
commons-beanutils-core-1.8.0.jar
and
-rw-r--r-- 1 jacek staff 445288 17 wrz 09:51 antlr-2.7.7.jar
-rw-r--r-- 1 jacek staff 164368 17 wrz 09:51 antlr-runtime-3.4.jar
-rw-r--r-- 1 jacek staff 302248 17 wrz 09:51 antlr4-runtime-4.5.3.jar
Even if that does not cause any class mismatches, it might be worth to
exclude them to minimize the size of the Spark distro.
What do you think?
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Different versions of dependencies in assembly/target/scala-2.11/jars?
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Sean,
"Boy it's a long story"...yeah, you tell me! :) I don't seem to find
anything worth reporting so...let's keep these possible discrepancies
in mind and be back to them when they hit us.
Thanks a lot, Sean. Your patience with dealing with people here and on
JIRA has always made me wish for having it. Kudos!
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Sun, Sep 18, 2016 at 5:14 PM, Sean Owen <so...@cloudera.com> wrote:
> Boy it's a long story, but I think the short answer is that it's only
> worth manually fixing the mismatches that are clearly going to cause a
> problem.
>
> Dependency version mismatch is inevitable, and Maven will always
> settle on one version of a particular group/artifact using a
> nearest-wins rule (SBT uses latest-wins). However it's possible that
> you get different versions of closely-related artifacts. Often it
> doesn't matter; sometimes it does.
>
> It's always possible to force a version of a group/artifact with
> <dependencyManagement>. The drawback is that, as dependencies evolve,
> you may be silently forcing that to an older version than other
> dependencies want. It builds up its own quiet legacy problem.
>
>
>
> On Sat, Sep 17, 2016 at 11:08 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>> Hi Sean,
>>
>> Thanks a lot for help understanding the different jars.
>>
>> Do you think there's anything that should be reported as an
>> enhancement/issue/task in JIRA?
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Sat, Sep 17, 2016 at 11:34 PM, Sean Owen <so...@cloudera.com> wrote:
>>> No, these are different major versions of these components, each of
>>> which gets used by something in the transitive dependency graph. They
>>> are not redundant because they're not actually presenting roughly the
>>> same component in the same namespace.
>>>
>>> However the parquet-hadoop bit looks wrong, in that it should be
>>> harmonized to one 1.x version. It's not that Spark uses inconsistent
>>> versions but that transitive deps do. We can still harmonize them in
>>> the build if it causes problems.
>>>
>>> On Sat, Sep 17, 2016 at 8:14 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>>>> Hi,
>>>>
>>>> Just noticed in assembly/target/scala-2.11/jars that similar libraries
>>>> have different versions:
>>>>
>>>> -rw-r--r-- 1 jacek staff 1230201 17 wrz 09:51 netty-3.8.0.Final.jar
>>>> -rw-r--r-- 1 jacek staff 2305335 17 wrz 09:51 netty-all-4.0.41.Final.jar
>>>>
>>>> and
>>>>
>>>> -rw-r--r-- 1 jacek staff 218076 17 wrz 09:51 parquet-hadoop-1.8.1.jar
>>>> -rw-r--r-- 1 jacek staff 2796935 17 wrz 09:51
>>>> parquet-hadoop-bundle-1.6.0.jar
>>>>
>>>> and
>>>>
>>>> -rw-r--r-- 1 jacek staff 46983 17 wrz 09:51 jackson-annotations-2.6.5.jar
>>>> -rw-r--r-- 1 jacek staff 258876 17 wrz 09:51 jackson-core-2.6.5.jar
>>>> -rw-r--r-- 1 jacek staff 232248 17 wrz 09:51 jackson-core-asl-1.9.13.jar
>>>> -rw-r--r-- 1 jacek staff 1171380 17 wrz 09:51 jackson-databind-2.6.5.jar
>>>> -rw-r--r-- 1 jacek staff 18336 17 wrz 09:51 jackson-jaxrs-1.9.13.jar
>>>> -rw-r--r-- 1 jacek staff 780664 17 wrz 09:51 jackson-mapper-asl-1.9.13.jar
>>>> -rw-r--r-- 1 jacek staff 41263 17 wrz 09:51
>>>> jackson-module-paranamer-2.6.5.jar
>>>> -rw-r--r-- 1 jacek staff 515604 17 wrz 09:51
>>>> jackson-module-scala_2.11-2.6.5.jar
>>>> -rw-r--r-- 1 jacek staff 27084 17 wrz 09:51 jackson-xc-1.9.13.jar
>>>>
>>>> and
>>>>
>>>> -rw-r--r-- 1 jacek staff 188671 17 wrz 09:51 commons-beanutils-1.7.0.jar
>>>> -rw-r--r-- 1 jacek staff 206035 17 wrz 09:51
>>>> commons-beanutils-core-1.8.0.jar
>>>>
>>>> and
>>>>
>>>> -rw-r--r-- 1 jacek staff 445288 17 wrz 09:51 antlr-2.7.7.jar
>>>> -rw-r--r-- 1 jacek staff 164368 17 wrz 09:51 antlr-runtime-3.4.jar
>>>> -rw-r--r-- 1 jacek staff 302248 17 wrz 09:51 antlr4-runtime-4.5.3.jar
>>>>
>>>> Even if that does not cause any class mismatches, it might be worth to
>>>> exclude them to minimize the size of the Spark distro.
>>>>
>>>> What do you think?
>>>>
>>>> Pozdrawiam,
>>>> Jacek Laskowski
>>>> ----
>>>> https://medium.com/@jaceklaskowski/
>>>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>>>> Follow me at https://twitter.com/jaceklaskowski
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Different versions of dependencies in assembly/target/scala-2.11/jars?
Posted by Sean Owen <so...@cloudera.com>.
Boy it's a long story, but I think the short answer is that it's only
worth manually fixing the mismatches that are clearly going to cause a
problem.
Dependency version mismatch is inevitable, and Maven will always
settle on one version of a particular group/artifact using a
nearest-wins rule (SBT uses latest-wins). However it's possible that
you get different versions of closely-related artifacts. Often it
doesn't matter; sometimes it does.
It's always possible to force a version of a group/artifact with
<dependencyManagement>. The drawback is that, as dependencies evolve,
you may be silently forcing that to an older version than other
dependencies want. It builds up its own quiet legacy problem.
On Sat, Sep 17, 2016 at 11:08 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi Sean,
>
> Thanks a lot for help understanding the different jars.
>
> Do you think there's anything that should be reported as an
> enhancement/issue/task in JIRA?
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Sat, Sep 17, 2016 at 11:34 PM, Sean Owen <so...@cloudera.com> wrote:
>> No, these are different major versions of these components, each of
>> which gets used by something in the transitive dependency graph. They
>> are not redundant because they're not actually presenting roughly the
>> same component in the same namespace.
>>
>> However the parquet-hadoop bit looks wrong, in that it should be
>> harmonized to one 1.x version. It's not that Spark uses inconsistent
>> versions but that transitive deps do. We can still harmonize them in
>> the build if it causes problems.
>>
>> On Sat, Sep 17, 2016 at 8:14 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>>> Hi,
>>>
>>> Just noticed in assembly/target/scala-2.11/jars that similar libraries
>>> have different versions:
>>>
>>> -rw-r--r-- 1 jacek staff 1230201 17 wrz 09:51 netty-3.8.0.Final.jar
>>> -rw-r--r-- 1 jacek staff 2305335 17 wrz 09:51 netty-all-4.0.41.Final.jar
>>>
>>> and
>>>
>>> -rw-r--r-- 1 jacek staff 218076 17 wrz 09:51 parquet-hadoop-1.8.1.jar
>>> -rw-r--r-- 1 jacek staff 2796935 17 wrz 09:51
>>> parquet-hadoop-bundle-1.6.0.jar
>>>
>>> and
>>>
>>> -rw-r--r-- 1 jacek staff 46983 17 wrz 09:51 jackson-annotations-2.6.5.jar
>>> -rw-r--r-- 1 jacek staff 258876 17 wrz 09:51 jackson-core-2.6.5.jar
>>> -rw-r--r-- 1 jacek staff 232248 17 wrz 09:51 jackson-core-asl-1.9.13.jar
>>> -rw-r--r-- 1 jacek staff 1171380 17 wrz 09:51 jackson-databind-2.6.5.jar
>>> -rw-r--r-- 1 jacek staff 18336 17 wrz 09:51 jackson-jaxrs-1.9.13.jar
>>> -rw-r--r-- 1 jacek staff 780664 17 wrz 09:51 jackson-mapper-asl-1.9.13.jar
>>> -rw-r--r-- 1 jacek staff 41263 17 wrz 09:51
>>> jackson-module-paranamer-2.6.5.jar
>>> -rw-r--r-- 1 jacek staff 515604 17 wrz 09:51
>>> jackson-module-scala_2.11-2.6.5.jar
>>> -rw-r--r-- 1 jacek staff 27084 17 wrz 09:51 jackson-xc-1.9.13.jar
>>>
>>> and
>>>
>>> -rw-r--r-- 1 jacek staff 188671 17 wrz 09:51 commons-beanutils-1.7.0.jar
>>> -rw-r--r-- 1 jacek staff 206035 17 wrz 09:51
>>> commons-beanutils-core-1.8.0.jar
>>>
>>> and
>>>
>>> -rw-r--r-- 1 jacek staff 445288 17 wrz 09:51 antlr-2.7.7.jar
>>> -rw-r--r-- 1 jacek staff 164368 17 wrz 09:51 antlr-runtime-3.4.jar
>>> -rw-r--r-- 1 jacek staff 302248 17 wrz 09:51 antlr4-runtime-4.5.3.jar
>>>
>>> Even if that does not cause any class mismatches, it might be worth to
>>> exclude them to minimize the size of the Spark distro.
>>>
>>> What do you think?
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> ----
>>> https://medium.com/@jaceklaskowski/
>>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>>> Follow me at https://twitter.com/jaceklaskowski
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Different versions of dependencies in assembly/target/scala-2.11/jars?
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Sean,
Thanks a lot for help understanding the different jars.
Do you think there's anything that should be reported as an
enhancement/issue/task in JIRA?
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Sat, Sep 17, 2016 at 11:34 PM, Sean Owen <so...@cloudera.com> wrote:
> No, these are different major versions of these components, each of
> which gets used by something in the transitive dependency graph. They
> are not redundant because they're not actually presenting roughly the
> same component in the same namespace.
>
> However the parquet-hadoop bit looks wrong, in that it should be
> harmonized to one 1.x version. It's not that Spark uses inconsistent
> versions but that transitive deps do. We can still harmonize them in
> the build if it causes problems.
>
> On Sat, Sep 17, 2016 at 8:14 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>> Hi,
>>
>> Just noticed in assembly/target/scala-2.11/jars that similar libraries
>> have different versions:
>>
>> -rw-r--r-- 1 jacek staff 1230201 17 wrz 09:51 netty-3.8.0.Final.jar
>> -rw-r--r-- 1 jacek staff 2305335 17 wrz 09:51 netty-all-4.0.41.Final.jar
>>
>> and
>>
>> -rw-r--r-- 1 jacek staff 218076 17 wrz 09:51 parquet-hadoop-1.8.1.jar
>> -rw-r--r-- 1 jacek staff 2796935 17 wrz 09:51
>> parquet-hadoop-bundle-1.6.0.jar
>>
>> and
>>
>> -rw-r--r-- 1 jacek staff 46983 17 wrz 09:51 jackson-annotations-2.6.5.jar
>> -rw-r--r-- 1 jacek staff 258876 17 wrz 09:51 jackson-core-2.6.5.jar
>> -rw-r--r-- 1 jacek staff 232248 17 wrz 09:51 jackson-core-asl-1.9.13.jar
>> -rw-r--r-- 1 jacek staff 1171380 17 wrz 09:51 jackson-databind-2.6.5.jar
>> -rw-r--r-- 1 jacek staff 18336 17 wrz 09:51 jackson-jaxrs-1.9.13.jar
>> -rw-r--r-- 1 jacek staff 780664 17 wrz 09:51 jackson-mapper-asl-1.9.13.jar
>> -rw-r--r-- 1 jacek staff 41263 17 wrz 09:51
>> jackson-module-paranamer-2.6.5.jar
>> -rw-r--r-- 1 jacek staff 515604 17 wrz 09:51
>> jackson-module-scala_2.11-2.6.5.jar
>> -rw-r--r-- 1 jacek staff 27084 17 wrz 09:51 jackson-xc-1.9.13.jar
>>
>> and
>>
>> -rw-r--r-- 1 jacek staff 188671 17 wrz 09:51 commons-beanutils-1.7.0.jar
>> -rw-r--r-- 1 jacek staff 206035 17 wrz 09:51
>> commons-beanutils-core-1.8.0.jar
>>
>> and
>>
>> -rw-r--r-- 1 jacek staff 445288 17 wrz 09:51 antlr-2.7.7.jar
>> -rw-r--r-- 1 jacek staff 164368 17 wrz 09:51 antlr-runtime-3.4.jar
>> -rw-r--r-- 1 jacek staff 302248 17 wrz 09:51 antlr4-runtime-4.5.3.jar
>>
>> Even if that does not cause any class mismatches, it might be worth to
>> exclude them to minimize the size of the Spark distro.
>>
>> What do you think?
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Different versions of dependencies in assembly/target/scala-2.11/jars?
Posted by Sean Owen <so...@cloudera.com>.
No, these are different major versions of these components, each of
which gets used by something in the transitive dependency graph. They
are not redundant because they're not actually presenting roughly the
same component in the same namespace.
However the parquet-hadoop bit looks wrong, in that it should be
harmonized to one 1.x version. It's not that Spark uses inconsistent
versions but that transitive deps do. We can still harmonize them in
the build if it causes problems.
On Sat, Sep 17, 2016 at 8:14 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> Just noticed in assembly/target/scala-2.11/jars that similar libraries
> have different versions:
>
> -rw-r--r-- 1 jacek staff 1230201 17 wrz 09:51 netty-3.8.0.Final.jar
> -rw-r--r-- 1 jacek staff 2305335 17 wrz 09:51 netty-all-4.0.41.Final.jar
>
> and
>
> -rw-r--r-- 1 jacek staff 218076 17 wrz 09:51 parquet-hadoop-1.8.1.jar
> -rw-r--r-- 1 jacek staff 2796935 17 wrz 09:51
> parquet-hadoop-bundle-1.6.0.jar
>
> and
>
> -rw-r--r-- 1 jacek staff 46983 17 wrz 09:51 jackson-annotations-2.6.5.jar
> -rw-r--r-- 1 jacek staff 258876 17 wrz 09:51 jackson-core-2.6.5.jar
> -rw-r--r-- 1 jacek staff 232248 17 wrz 09:51 jackson-core-asl-1.9.13.jar
> -rw-r--r-- 1 jacek staff 1171380 17 wrz 09:51 jackson-databind-2.6.5.jar
> -rw-r--r-- 1 jacek staff 18336 17 wrz 09:51 jackson-jaxrs-1.9.13.jar
> -rw-r--r-- 1 jacek staff 780664 17 wrz 09:51 jackson-mapper-asl-1.9.13.jar
> -rw-r--r-- 1 jacek staff 41263 17 wrz 09:51
> jackson-module-paranamer-2.6.5.jar
> -rw-r--r-- 1 jacek staff 515604 17 wrz 09:51
> jackson-module-scala_2.11-2.6.5.jar
> -rw-r--r-- 1 jacek staff 27084 17 wrz 09:51 jackson-xc-1.9.13.jar
>
> and
>
> -rw-r--r-- 1 jacek staff 188671 17 wrz 09:51 commons-beanutils-1.7.0.jar
> -rw-r--r-- 1 jacek staff 206035 17 wrz 09:51
> commons-beanutils-core-1.8.0.jar
>
> and
>
> -rw-r--r-- 1 jacek staff 445288 17 wrz 09:51 antlr-2.7.7.jar
> -rw-r--r-- 1 jacek staff 164368 17 wrz 09:51 antlr-runtime-3.4.jar
> -rw-r--r-- 1 jacek staff 302248 17 wrz 09:51 antlr4-runtime-4.5.3.jar
>
> Even if that does not cause any class mismatches, it might be worth to
> exclude them to minimize the size of the Spark distro.
>
> What do you think?
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org