You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/05/12 19:10:31 UTC
spark git commit: [SPARK-14421] Upgrades protobuf dependency to 2.6.1 for the new version of KCL, and…
Repository: spark
Updated Branches:
refs/heads/master 81bf87084 -> 81e3bfc16
[SPARK-14421] Upgrades protobuf dependency to 2.6.1 for the new version of KCL, and\u2026
## What changes were proposed in this pull request?
When running with Kinesis Consumer Library (KCL), against a stream that contains aggregated data, the KCL needs access to protobuf to de-aggregate the records. Without this patch, that results in the following error message:
```
Caused by: java.lang.ClassNotFoundException: com.google.protobuf.ProtocolStringList
```
This PR upgrades the protobuf dependency within the kinesis-asl-assembly, and relocates that package (as not to conflict with Spark's use of 2.5.0), which fixes the above CNFE.
## How was this patch tested?
Used kinesis word count example against a stream containing aggregated data.
See: SPARK-14421
Author: Brian O'Neill <bo...@alumni.brown.edu>
Closes #13054 from boneill42/protobuf-relocation-for-kcl.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/81e3bfc1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/81e3bfc1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/81e3bfc1
Branch: refs/heads/master
Commit: 81e3bfc16c6cfbf2f9f2c6c32ed651b8450795ba
Parents: 81bf870
Author: Brian O'Neill <bo...@alumni.brown.edu>
Authored: Thu May 12 20:10:33 2016 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Thu May 12 20:10:33 2016 +0100
----------------------------------------------------------------------
external/kinesis-asl-assembly/pom.xml | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/81e3bfc1/external/kinesis-asl-assembly/pom.xml
----------------------------------------------------------------------
diff --git a/external/kinesis-asl-assembly/pom.xml b/external/kinesis-asl-assembly/pom.xml
index e057b78..6fb88eb 100644
--- a/external/kinesis-asl-assembly/pom.xml
+++ b/external/kinesis-asl-assembly/pom.xml
@@ -63,7 +63,12 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
- <scope>provided</scope>
+ <version>2.6.1</version>
+ <!--
+ We are being explicit about version here and overriding the
+ spark default of 2.5.0 because KCL appears to have introduced
+ a dependency on protobuf 2.6.1 somewhere between KCL 1.4.0 and 1.6.1.
+ -->
</dependency>
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
@@ -147,6 +152,15 @@
<include>*:*</include>
</includes>
</artifactSet>
+ <relocations>
+ <relocation>
+ <pattern>com.google.protobuf</pattern>
+ <shadedPattern>kinesis.protobuf</shadedPattern>
+ <includes>
+ <include>com.google.protobuf.**</include>
+ </includes>
+ </relocation>
+ </relocations>
<filters>
<filter>
<artifact>*:*</artifact>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org