You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Bryan Cutler (JIRA)" <ji...@apache.org> on 2017/12/05 17:27:00 UTC
[jira] [Comment Edited] (ARROW-1818) Examine Java Dependencies
[ https://issues.apache.org/jira/browse/ARROW-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278897#comment-16278897 ]
Bryan Cutler edited comment on ARROW-1818 at 12/5/17 5:26 PM:
--------------------------------------------------------------
After running the latest snapshot through Spark and checking dependencies, everything looks pretty good. It does add netty artifacts that weren't there before, I believe they were not properly declared in Arrow in 0.4 and that was changed by this https://github.com/apache/arrow/pull/978. I think it should be fine for Spark as they could probably just be excluded since "netty-all" is already there. cc [~icexelloss] [~holdenk]
{noformat}
diff --git a/dev/deps/spark-deps-hadoop-2.7 b/dev/deps/spark-deps-hadoop-2.7
index cde9a5e..29bb251 100644
--- a/dev/deps/spark-deps-hadoop-2.7
+++ b/dev/deps/spark-deps-hadoop-2.7
@@ -14,9 +14,9 @@ apacheds-kerberos-codec-2.0.0-M15.jar
api-asn1-api-1.0.0-M20.jar
api-util-1.0.0-M20.jar
arpack_combined_all-0.1.jar
-arrow-format-0.4.0.jar
-arrow-memory-0.4.0.jar
-arrow-vector-0.4.0.jar
+arrow-format-0.8.0-SNAPSHOT.jar
+arrow-memory-0.8.0-SNAPSHOT.jar
+arrow-vector-0.8.0-SNAPSHOT.jar
avro-1.7.7.jar
avro-ipc-1.7.7.jar
avro-mapred-1.7.7-hadoop2.jar
@@ -82,7 +82,7 @@ hadoop-yarn-server-web-proxy-2.7.3.jar
hk2-api-2.4.0-b34.jar
hk2-locator-2.4.0-b34.jar
hk2-utils-2.4.0-b34.jar
-hppc-0.7.1.jar
+hppc-0.7.2.jar
htrace-core-3.1.0-incubating.jar
httpclient-4.5.2.jar
httpcore-4.4.4.jar
@@ -146,6 +146,8 @@ metrics-jvm-3.1.5.jar
minlog-1.3.0.jar
netty-3.9.9.Final.jar
netty-all-4.1.17.Final.jar
+netty-buffer-4.1.17.Final.jar
+netty-common-4.1.17.Final.jar
objenesis-2.1.jar
{noformat}
was (Author: bryanc):
After running the latest snapshot through Spark and checking dependencies, everything looks pretty good. It does add netty artifacts that weren't there before, I believe they were not properly declared in Arrow and that was changed by this https://github.com/apache/arrow/pull/978. I think it should be fine for Spark as they could probably just be excluded since "netty-all" is already there. cc [~icexelloss] [~holdenk]
{noformat}
diff --git a/dev/deps/spark-deps-hadoop-2.7 b/dev/deps/spark-deps-hadoop-2.7
index cde9a5e..29bb251 100644
--- a/dev/deps/spark-deps-hadoop-2.7
+++ b/dev/deps/spark-deps-hadoop-2.7
@@ -14,9 +14,9 @@ apacheds-kerberos-codec-2.0.0-M15.jar
api-asn1-api-1.0.0-M20.jar
api-util-1.0.0-M20.jar
arpack_combined_all-0.1.jar
-arrow-format-0.4.0.jar
-arrow-memory-0.4.0.jar
-arrow-vector-0.4.0.jar
+arrow-format-0.8.0-SNAPSHOT.jar
+arrow-memory-0.8.0-SNAPSHOT.jar
+arrow-vector-0.8.0-SNAPSHOT.jar
avro-1.7.7.jar
avro-ipc-1.7.7.jar
avro-mapred-1.7.7-hadoop2.jar
@@ -82,7 +82,7 @@ hadoop-yarn-server-web-proxy-2.7.3.jar
hk2-api-2.4.0-b34.jar
hk2-locator-2.4.0-b34.jar
hk2-utils-2.4.0-b34.jar
-hppc-0.7.1.jar
+hppc-0.7.2.jar
htrace-core-3.1.0-incubating.jar
httpclient-4.5.2.jar
httpcore-4.4.4.jar
@@ -146,6 +146,8 @@ metrics-jvm-3.1.5.jar
minlog-1.3.0.jar
netty-3.9.9.Final.jar
netty-all-4.1.17.Final.jar
+netty-buffer-4.1.17.Final.jar
+netty-common-4.1.17.Final.jar
objenesis-2.1.jar
{noformat}
> Examine Java Dependencies
> -------------------------
>
> Key: ARROW-1818
> URL: https://issues.apache.org/jira/browse/ARROW-1818
> Project: Apache Arrow
> Issue Type: Task
> Components: Java - Vectors
> Reporter: Bryan Cutler
> Assignee: Bryan Cutler
> Fix For: 0.8.0
>
>
> While integrating the latest Arrow Java with Spark master, I noticed some possible binary incompatibilities with dependencies. I'd like to examine these a little closer and make sure there are no problems before 0.8 is cut.
> {noformat}
> Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
> [warn]
> [warn] * com.google.code.findbugs:jsr305:3.0.2 is selected over {1.3.9, 3.0.0}
> [warn] +- org.apache.arrow:arrow-vector:0.8.0-SNAPSHOT (depends on 3.0.2)
> [warn] +- org.apache.arrow:arrow-memory:0.8.0-SNAPSHOT (depends on 3.0.2)
> [warn] +- org.apache.hadoop:hadoop-common:2.7.3 (depends on 3.0.0)
> [warn] +- org.apache.spark:spark-unsafe_2.11:2.3.0-SNAPSHOT (depends on 1.3.9)
> [warn] +- org.apache.spark:spark-core_2.11:2.3.0-SNAPSHOT (depends on 1.3.9)
> [warn] +- org.apache.spark:spark-network-common_2.11:2.3.0-SNAPSHOT (depends on 1.3.9)
> [warn]
> [warn] * io.netty:netty:3.9.9.Final is selected over {3.6.2.Final, 3.7.0.Final}
> [warn] +- org.apache.spark:spark-core_2.11:2.3.0-SNAPSHOT (depends on 3.9.9.Final)
> [warn] +- org.apache.hadoop:hadoop-hdfs:2.7.3 (depends on 3.6.2.Final)
> [warn] +- org.apache.zookeeper:zookeeper:3.4.6 (depends on 3.6.2.Final)
> [warn]
> [warn] * io.netty:netty-all:4.0.47.Final is selected over 4.0.23.Final
> [warn] +- org.apache.hadoop:hadoop-hdfs:2.7.3 (depends on 4.0.23.Final)
> [warn] +- org.apache.spark:spark-core_2.11:2.3.0-SNAPSHOT (depends on 4.0.23.Final)
> [warn] +- org.apache.spark:spark-network-common_2.11:2.3.0-SNAPSHOT (depends on 4.0.23.Final)
> [warn]
> [warn] * commons-net:commons-net:3.1 is selected over 2.2
> [warn] +- org.apache.hadoop:hadoop-common:2.7.3 (depends on 3.1)
> [warn] +- org.apache.spark:spark-core_2.11:2.3.0-SNAPSHOT (depends on 2.2)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)