You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/01/06 00:15:13 UTC
spark git commit: [SPARK-12453][STREAMING] Remove explicit dependency
on aws-java-sdk
Repository: spark
Updated Branches:
refs/heads/master 78015a8b7 -> ff8997554
[SPARK-12453][STREAMING] Remove explicit dependency on aws-java-sdk
Successfully ran kinesis demo on a live, aws hosted kinesis stream against master and 1.6 branches. For reasons I don't entirely understand it required a manual merge to 1.5 which I did as shown here: https://github.com/BrianLondon/spark/commit/075c22e89bc99d5e99be21f40e0d72154a1e23a2
The demo ran successfully on the 1.5 branch as well.
According to `mvn dependency:tree` it is still pulling a fairly old version of the aws-java-sdk (1.9.37), but this appears to have fixed the kinesis regression in 1.5.2.
Author: BrianLondon <br...@seatgeek.com>
Closes #10492 from BrianLondon/remove-only.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ff899755
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ff899755
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ff899755
Branch: refs/heads/master
Commit: ff89975543b153d0d235c0cac615d45b34aa8fe7
Parents: 78015a8
Author: BrianLondon <br...@seatgeek.com>
Authored: Tue Jan 5 23:15:07 2016 +0000
Committer: Sean Owen <so...@cloudera.com>
Committed: Tue Jan 5 23:15:07 2016 +0000
----------------------------------------------------------------------
extras/kinesis-asl/pom.xml | 5 -----
.../org/apache/spark/streaming/kinesis/KinesisReceiver.scala | 1 +
pom.xml | 1 -
3 files changed, 1 insertion(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/ff899755/extras/kinesis-asl/pom.xml
----------------------------------------------------------------------
diff --git a/extras/kinesis-asl/pom.xml b/extras/kinesis-asl/pom.xml
index 3c57225..20e2c5e 100644
--- a/extras/kinesis-asl/pom.xml
+++ b/extras/kinesis-asl/pom.xml
@@ -61,11 +61,6 @@
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
- <artifactId>aws-java-sdk</artifactId>
- <version>${aws.java.sdk.version}</version>
- </dependency>
- <dependency>
- <groupId>com.amazonaws</groupId>
<artifactId>amazon-kinesis-producer</artifactId>
<version>${aws.kinesis.producer.version}</version>
<scope>test</scope>
http://git-wip-us.apache.org/repos/asf/spark/blob/ff899755/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala
----------------------------------------------------------------------
diff --git a/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala b/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala
index 80edda5..abb9b6c 100644
--- a/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala
+++ b/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala
@@ -185,6 +185,7 @@ private[kinesis] class KinesisReceiver[T](
workerThread.setName(s"Kinesis Receiver ${streamId}")
workerThread.setDaemon(true)
workerThread.start()
+
logInfo(s"Started receiver with workerId $workerId")
}
http://git-wip-us.apache.org/repos/asf/spark/blob/ff899755/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 398fcc9..d0ac1eb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -152,7 +152,6 @@
<avro.version>1.7.7</avro.version>
<avro.mapred.classifier>hadoop2</avro.mapred.classifier>
<jets3t.version>0.7.1</jets3t.version>
- <aws.java.sdk.version>1.9.40</aws.java.sdk.version>
<aws.kinesis.client.version>1.4.0</aws.kinesis.client.version>
<!-- the producer is used in tests -->
<aws.kinesis.producer.version>0.10.1</aws.kinesis.producer.version>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org