You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2017/02/08 12:21:57 UTC
spark git commit: [SPARK-19409][BUILD][TEST-MAVEN] Fix
ParquetAvroCompatibilitySuite failure due to test dependency on avro
Repository: spark
Updated Branches:
refs/heads/master e8d3fca45 -> 0077bfcb9
[SPARK-19409][BUILD][TEST-MAVEN] Fix ParquetAvroCompatibilitySuite failure due to test dependency on avro
## What changes were proposed in this pull request?
After using Apache Parquet 1.8.2, `ParquetAvroCompatibilitySuite` fails on **Maven** test. It is because `org.apache.parquet.avro.AvroParquetWriter` in the test code used new `avro 1.8.0` specific class, `LogicalType`. This PR aims to fix the test dependency of `sql/core` module to use avro 1.8.0.
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/2530/consoleFull
```
ParquetAvroCompatibilitySuite:
*** RUN ABORTED ***
java.lang.NoClassDefFoundError: org/apache/avro/LogicalType
at org.apache.parquet.avro.AvroParquetWriter.writeSupport(AvroParquetWriter.java:144)
```
## How was this patch tested?
Pass the existing test with **Maven**.
```
$ build/mvn -Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive -Phive-thriftserver test
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:07 h
[INFO] Finished at: 2017-02-04T05:41:43+00:00
[INFO] Final Memory: 77M/987M
[INFO] ------------------------------------------------------------------------
```
Author: Dongjoon Hyun <do...@apache.org>
Closes #16795 from dongjoon-hyun/SPARK-19409-2.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0077bfcb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0077bfcb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0077bfcb
Branch: refs/heads/master
Commit: 0077bfcb93832d93009f73f4b80f2e3d98fd2fa4
Parents: e8d3fca
Author: Dongjoon Hyun <do...@apache.org>
Authored: Wed Feb 8 12:21:49 2017 +0000
Committer: Sean Owen <so...@cloudera.com>
Committed: Wed Feb 8 12:21:49 2017 +0000
----------------------------------------------------------------------
resource-managers/mesos/pom.xml | 7 +++++++
sql/core/pom.xml | 13 +++++++++++++
2 files changed, 20 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/0077bfcb/resource-managers/mesos/pom.xml
----------------------------------------------------------------------
diff --git a/resource-managers/mesos/pom.xml b/resource-managers/mesos/pom.xml
index c0a8f9a..03846d9 100644
--- a/resource-managers/mesos/pom.xml
+++ b/resource-managers/mesos/pom.xml
@@ -49,6 +49,13 @@
</dependency>
<dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-tags_${scala.binary.version}</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
<groupId>org.apache.mesos</groupId>
<artifactId>mesos</artifactId>
<version>${mesos.version}</version>
http://git-wip-us.apache.org/repos/asf/spark/blob/0077bfcb/sql/core/pom.xml
----------------------------------------------------------------------
diff --git a/sql/core/pom.xml b/sql/core/pom.xml
index b8aa698..69d797b 100644
--- a/sql/core/pom.xml
+++ b/sql/core/pom.xml
@@ -129,6 +129,19 @@
<artifactId>parquet-avro</artifactId>
<scope>test</scope>
</dependency>
+ <!--
+ This version of avro test-dep is different from the one defined
+ in the parent pom. The parent pom has avro 1.7.7 test-dep for Hadoop.
+ Here, ParquetAvroCompatibilitySuite uses parquet-avro's AvroParquetWriter
+ which uses avro 1.8.0+ specific API. In Maven 3, we need to have
+ this here to have different versions for the same artifact.
+ -->
+ <dependency>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ <version>1.8.1</version>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org