You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cfregly <gi...@git.apache.org> on 2014/07/16 11:17:37 UTC
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
GitHub user cfregly opened a pull request:
https://github.com/apache/spark/pull/1434
[SPARK-1981] Add AWS Kinesis streaming support
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cfregly/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1434.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1434
----
commit b3b0ff118cac3c0a5a10f9912b383bb0665c9a1b
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-16T07:03:04Z
[SPARK-1981] Add AWS Kinesis streaming support
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15504399
--- Diff: dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala ---
@@ -47,7 +47,14 @@ object SimpleApp {
System.exit(-1)
}
if (foundGanglia) {
- println("Ganglia sink was loaded via spark-core")
+ println("Ganglia sink was loaded via spark-ganglia-lgpl")
+ System.exit(-1)
+ }
+
+ // Remove kinesis from default build due to ASL license issue
--- End diff --
Nice! Good that you added this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50250374
QA tests have started for PR 1434. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17230/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15555270
--- Diff: bin/run-kinesis-example ---
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+
--- End diff --
yup, yup. this is a special circumstance due to the licensing. i didn't want to complicate the examples build by depending on an optional package, so this was the best workaround.
i put the scripts alongside the other scripts to reduce confusion, but i can move it to extras/kinesis/bin for sure. i'll update the docs accordingly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50792452
QA tests have started for PR 1434. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17589/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15618546
--- Diff: extras/spark-kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCount.java ---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.examples.streaming;
+
+import java.util.List;
+import java.util.regex.Pattern;
+
+import org.apache.log4j.Level;
+import org.apache.log4j.Logger;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.api.java.function.Function2;
+import org.apache.spark.api.java.function.PairFunction;
+import org.apache.spark.storage.StorageLevel;
+import org.apache.spark.streaming.Duration;
+import org.apache.spark.streaming.Milliseconds;
+import org.apache.spark.streaming.api.java.JavaDStream;
+import org.apache.spark.streaming.api.java.JavaPairDStream;
+import org.apache.spark.streaming.api.java.JavaStreamingContext;
+import org.apache.spark.streaming.dstream.DStream;
+import org.apache.spark.streaming.kinesis.KinesisRecordSerializer;
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer;
+import org.apache.spark.streaming.kinesis.KinesisUtils;
+
+import scala.Tuple2;
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.kinesis.AmazonKinesisClient;
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
+import com.google.common.base.Optional;
+import com.google.common.collect.Lists;
+
+/**
+ * Java-friendly Kinesis Spark Streaming WordCount example
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the given stream.
+ * It then starts pulling from the tip of the given <stream-name> and <endpoint-url> at the given <batch-interval>.
+ * Because we're pulling from the tip (InitialPositionInStream.LATEST), only new stream data will be picked up after the KinesisReceiver starts.
+ * This could lead to missed records if data is added to the stream while no KinesisReceivers are running.
+ * In production, you'll want to switch to InitialPositionInStream.TRIM_HORIZON which will read up to 24 hours (Kinesis limit) of previous stream data
+ * depending on the checkpoint frequency.
+ * InitialPositionInStream.TRIM_HORIZON may lead to duplicate processing of records depending on the checkpoint frequency.
+ * Record processing should be idempotent when possible.
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: JavaKinesisWordCount <stream-name> <endpoint-url> <batch-interval>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service (ie. https://kinesis.us-east-1.amazonaws.com)
+ * <batch-interval> is the batch interval in milliseconds (ie. 1000ms)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ bin/run-kinesis-example \
+ * org.apache.spark.examples.streaming.JavaKinesisWordCount mySparkStream https://kinesis.us-east-1.amazonaws.com 1000
+ *
+ * There is a companion helper class called KinesisWordCountProducer which puts dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in the class definition.
+ */
+public final class JavaKinesisWordCount {
+ private static final Pattern WORD_SEPARATOR = Pattern.compile(" ");
+ private static final Logger logger = Logger.getLogger(JavaKinesisWordCount.class);
+
+ /**
+ * Make the constructor private to enforce singleton
+ */
+ private JavaKinesisWordCount() {
+ }
+
+ public static void main(String[] args) {
+ /**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 3) {
+ System.err.println("Usage: JavaKinesisWordCount <stream-name> <kinesis-endpoint-url> <batch-interval>");
+ System.exit(1);
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ boolean log4jInitialized = Logger.getRootLogger().getAllAppenders()
+ .hasMoreElements();
+ if (!log4jInitialized) {
+ /** We first log something to initialize Spark's default logging, then we override the logging level. */
+ Logger.getRootLogger()
+ .info("Setting log level to [ERROR] for streaming example."
+ + " To override add a custom log4j.properties to the classpath.");
+ Logger.getRootLogger().setLevel(Level.ERROR);
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ String stream = args[0];
+ String endpoint = args[1];
+ Integer batchIntervalMillis = Integer.valueOf(args[2]);
--- End diff --
fixed at 2 seconds. i kept the dependent intervals as multiples of this batch interval because i think this is important to enforce. i'm no longer relying on ther $times() method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15564321
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): ReceiverInputDStream[Array[Byte]] = {
+
+ ssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return JavaReceiverInputDStream[Array[Byte]]
+ */
+ def createJavaStream(
+ jssc: JavaStreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): JavaReceiverInputDStream[Array[Byte]] = {
+
+ jssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create checkpoint state using the existing system clock
+ * @param checkpointIntervalMillis
+ */
+ def createCheckpointState(checkpointIntervalMillis: Long): CheckpointState = {
+ new CheckpointState(checkpointIntervalMillis)
+ }
+
+ /**
+ * Retry the given amount of times with a random backoff time (millis) less than the given maxBackOffMillis
+ *
+ * @param expression expression to evalute
+ * @param numRetriesLeft number of retries left
+ * @param maxBackOffMillis: max millis between retries
+ *
+ * @return Evaluation of the given expression
+ * @throws Unretryable exception, unexpected exception,
+ * or any exception that persists after numRetriesLeft reaches 0
+ */
+ @annotation.tailrec
+ def retry[T](expression: => T, numRetriesLeft: Int, maxBackOffMillis: Int): T = {
--- End diff --
Actually, I correct myself. It is being used in the KinesisReceiver. However, this should not be exposed to the user. In fact, its better to keep this user facing class clean, and put this in the KinesisReceiver object, as that is the main user of retry.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by pdeyhim <gi...@git.apache.org>.
Github user pdeyhim commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50974841
Awsome work!! @tdas @cfregly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15724923
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.util.List
+
+import scala.collection.JavaConversions.asScalaBuffer
+import scala.util.Random
+
+import org.apache.spark.Logging
+
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer
+import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownReason
+import com.amazonaws.services.kinesis.model.Record
+
+/**
+ * Kinesis-specific implementation of the Kinesis Client Library (KCL) IRecordProcessor.
+ * This implementation operates on the Array[Byte] from the KinesisReceiver.
+ * The Kinesis Worker creates an instance of this KinesisRecordProcessor upon startup.
+ *
+ * @param receiver Kinesis receiver
+ * @param workerId for logging purposes
+ * @param checkpointState represents the checkpoint state including the next checkpoint time.
+ * It's injected here for mocking purposes.
+ */
+private[kinesis] class KinesisRecordProcessor(
+ receiver: KinesisReceiver,
+ workerId: String,
+ checkpointState: KinesisCheckpointState) extends IRecordProcessor with Logging {
+
+ /** shardId to be populated during initialize() */
+ var shardId: String = _
+
+ /**
+ * The Kinesis Client Library calls this method during IRecordProcessor initialization.
+ *
+ * @param shardId assigned by the KCL to this particular RecordProcessor.
+ */
+ override def initialize(shardId: String) {
+ logInfo(s"Initialize: Initializing workerId $workerId with shardId $shardId")
+ this.shardId = shardId
+ }
+
+ /**
+ * This method is called by the KCL when a batch of records is pulled from the Kinesis stream.
+ * This is the record-processing bridge between the KCL's IRecordProcessor.processRecords()
+ * and Spark Streaming's Receiver.store().
+ *
+ * @param batch list of records from the Kinesis stream shard
+ * @param checkpointer used to update Kinesis when this batch has been processed/stored
+ * in the DStream
+ */
+ override def processRecords(batch: List[Record], checkpointer: IRecordProcessorCheckpointer) {
+ if (!receiver.isStopped()) {
+ try {
+ /**
+ * Note: If we try to store the raw ByteBuffer from record.getData(), the Spark Streaming
+ * Receiver.store(ByteBuffer) attempts to deserialize the ByteBuffer using the
+ * internally-configured Spark serializer (kryo, etc).
+ * This is not desirable, so we instead store a raw Array[Byte] and decouple
+ * ourselves from Spark's internal serialization strategy.
+ */
+ batch.foreach(record =>
+ KinesisRecordProcessor.retry(receiver.store(record.getData().array()), 4, 500)
--- End diff --
Backing off by 500 here is a probably a bad idea. Maybe reduce it to 10 ms.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50253442
this PR worked for @srosenthal , btw.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50511485
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15555598
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+import java.nio.ByteBuffer
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.
+ *
+ * @param app name
+ * @param Kinesis stream name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * @param persistence strategy for RDDs and DStreams.
+ */
+private[streaming] class KinesisReceiver(
+ app: String,
--- End diff --
good catch. i went through and fixed another similar formatting mistake, as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15726872
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Duration
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Helper class to create Amazon Kinesis Input Stream
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
--- End diff --
This does not need to extend Logging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15623426
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param appName Kinesis Application Name. Kinesis Apps are mapped to Kinesis Streams
+ * by the Kinesis Client Library. If you change the App name or Stream name,
+ * the KCL will throw errors.
+ * @param stream Kinesis Stream Name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types
+ * of checkpoints.
+ * @param initialPositionInStream in the absence of Kinesis checkpoint info, this is the
+ * worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours
+ * (InitialPositionInStream.TRIM_HORIZON) or the tip of the stream
+ * (InitialPositionInStream.LATEST).
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk
--- End diff --
This is wrong now as there is not default. Why was the default values removed? If there was issues with Java compatibility then its good to create alternate versions of createStream, with less number of parameters (so basically defining default parameters explicitly).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1434
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15504437
--- Diff: docs/streaming-custom-receivers.md ---
@@ -174,7 +174,7 @@ val words = lines.flatMap(_.split(" "))
...
{% endhighlight %}
-The full source code is in the example [CustomReceiver.scala](https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/streaming/examples/CustomReceiver.scala).
+The full source code is in the example [CustomReceiver.scala](https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/CustomReceiver.scala).
--- End diff --
Good catch!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50951626
QA tests have started for PR 1434. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17736/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15617770
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): ReceiverInputDStream[Array[Byte]] = {
+
+ ssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15668154
--- Diff: examples/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCount.scala ---
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming
+
+import java.nio.ByteBuffer
+
+import scala.util.Random
+
+import org.apache.log4j.Level
+import org.apache.log4j.Logger
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkContext.rddToOrderedRDDFunctions
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer
+import org.apache.spark.streaming.kinesis.KinesisUtils
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.model.PutRecordRequest
+
+/**
+ * Kinesis Spark Streaming WordCount example.
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on
+ * the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the
+ * given stream.
+ * It then starts pulling from the last checkpointed sequence number of the given
+ * <stream-name> and <endpoint-url>.
+ *
+ * Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials
+ * in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: KinesisWordCount <stream-name> <endpoint-url>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service
+ * (ie. https://kinesis.us-east-1.amazonaws.com)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ $SPARK_HOME/bin/run-example \
+ * org.apache.spark.examples.streaming.KinesisWordCount mySparkStream \
+ * https://kinesis.us-east-1.amazonaws.com
+ *
+ * There is a companion helper class below called KinesisWordCountProducer which puts
+ * dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in that class definition.
+ */
+object KinesisWordCount extends Logging {
+ val WordSeparator = " "
+
+ def main(args: Array[String]) {
+/**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 2) {
+ System.err.println("Usage: KinesisWordCount <stream-name> <endpoint-url>")
+ System.exit(1)
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency
+ * on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements()
+ if (!log4jInitialized) {
+ /**
+ * We first log something to initialize Spark's default logging,
+ * then we override the logging level.
+ * */
+ logInfo("Setting log level to [INFO] for streaming example." +
+ " To override add a custom log4j.properties to the classpath.")
+
+ Logger.getRootLogger().setLevel(Level.INFO)
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ val Array(stream, endpoint) = args
+ val batchIntervalMillis = 2000
+
+ /** Create a Kinesis client in order to determine the number of shards for the given stream */
+ val kinesisClient = new AmazonKinesisClient(new DefaultAWSCredentialsProviderChain())
+ kinesisClient.setEndpoint(endpoint)
+
+ /** Determine the number of shards from the stream */
+ val numShards = kinesisClient.describeStream(stream).getStreamDescription().getShards().size()
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard.*/
+ val numStreams = numShards
+
+ /**
+ * Must add 1 more thread than the number of receivers or the output won't show properly
+ * from the driver
+ */
+ val numSparkThreads = numStreams + 1
+
+ /** Set the app name */
+ val appName = "KinesisWordCount"
+
+ /** Setup the Spark config. */
+ val sparkConfig = new SparkConf().setAppName(appName).setMaster(s"local[$numSparkThreads]")
+
+ /**
+ * Set the batch interval.
+ * Records will be pulled from the Kinesis stream and stored as a single DStream within Spark
+ * every batch interval.
+ */
+ val batchInterval = Milliseconds(batchIntervalMillis)
+
+ /**
+ * It's recommended that you perform a Spark checkpoint between 5 and 10 times the batch
+ * interval.
+ * While this is the Spark checkpoint interval, we're going to use it for the Kinesis
+ * checkpoint interval, as well.
+ * For example purposes, we'll just use the batchInterval.
+ */
+ val checkpointInterval = batchInterval
+
+ /** Setup the StreamingContext */
+ val ssc = new StreamingContext(sparkConfig, batchInterval)
+
+ /** Setup the checkpoint directory used by Spark Streaming */
+ ssc.checkpoint("/tmp/checkpoint");
+
+ /**
+ * Create the same number of Kinesis Receivers/DStreams as stream shards, then union
+ * them all.
+ */
+ var allStreams: DStream[Array[Byte]] = KinesisUtils.createStream(ssc, appName, stream,
+ endpoint, checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the checkpoint interval */
+ allStreams.checkpoint(checkpointInterval)
+ for (i <- 1 until numStreams) {
+ /** Create a new Receiver/DStream for each stream shard */
+ val dStream = KinesisUtils.createStream(ssc, appName, stream, endpoint,
+ checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the Spark checkpoint interval */
+ dStream.checkpoint(checkpointInterval)
--- End diff --
Does this stream need to be checkpointed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15668266
--- Diff: examples/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCount.scala ---
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming
+
+import java.nio.ByteBuffer
+
+import scala.util.Random
+
+import org.apache.log4j.Level
+import org.apache.log4j.Logger
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkContext.rddToOrderedRDDFunctions
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer
+import org.apache.spark.streaming.kinesis.KinesisUtils
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.model.PutRecordRequest
+
+/**
+ * Kinesis Spark Streaming WordCount example.
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on
+ * the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the
+ * given stream.
+ * It then starts pulling from the last checkpointed sequence number of the given
+ * <stream-name> and <endpoint-url>.
+ *
+ * Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials
+ * in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: KinesisWordCount <stream-name> <endpoint-url>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service
+ * (ie. https://kinesis.us-east-1.amazonaws.com)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ $SPARK_HOME/bin/run-example \
+ * org.apache.spark.examples.streaming.KinesisWordCount mySparkStream \
+ * https://kinesis.us-east-1.amazonaws.com
+ *
+ * There is a companion helper class below called KinesisWordCountProducer which puts
+ * dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in that class definition.
+ */
+object KinesisWordCount extends Logging {
+ val WordSeparator = " "
+
+ def main(args: Array[String]) {
+/**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 2) {
+ System.err.println("Usage: KinesisWordCount <stream-name> <endpoint-url>")
+ System.exit(1)
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency
+ * on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements()
+ if (!log4jInitialized) {
+ /**
+ * We first log something to initialize Spark's default logging,
+ * then we override the logging level.
+ * */
+ logInfo("Setting log level to [INFO] for streaming example." +
+ " To override add a custom log4j.properties to the classpath.")
+
+ Logger.getRootLogger().setLevel(Level.INFO)
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ val Array(stream, endpoint) = args
+ val batchIntervalMillis = 2000
+
+ /** Create a Kinesis client in order to determine the number of shards for the given stream */
+ val kinesisClient = new AmazonKinesisClient(new DefaultAWSCredentialsProviderChain())
+ kinesisClient.setEndpoint(endpoint)
+
+ /** Determine the number of shards from the stream */
+ val numShards = kinesisClient.describeStream(stream).getStreamDescription().getShards().size()
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard.*/
+ val numStreams = numShards
+
+ /**
+ * Must add 1 more thread than the number of receivers or the output won't show properly
+ * from the driver
+ */
+ val numSparkThreads = numStreams + 1
+
+ /** Set the app name */
+ val appName = "KinesisWordCount"
+
+ /** Setup the Spark config. */
+ val sparkConfig = new SparkConf().setAppName(appName).setMaster(s"local[$numSparkThreads]")
+
+ /**
+ * Set the batch interval.
+ * Records will be pulled from the Kinesis stream and stored as a single DStream within Spark
+ * every batch interval.
+ */
+ val batchInterval = Milliseconds(batchIntervalMillis)
+
+ /**
+ * It's recommended that you perform a Spark checkpoint between 5 and 10 times the batch
+ * interval.
+ * While this is the Spark checkpoint interval, we're going to use it for the Kinesis
+ * checkpoint interval, as well.
+ * For example purposes, we'll just use the batchInterval.
+ */
+ val checkpointInterval = batchInterval
+
+ /** Setup the StreamingContext */
+ val ssc = new StreamingContext(sparkConfig, batchInterval)
+
+ /** Setup the checkpoint directory used by Spark Streaming */
+ ssc.checkpoint("/tmp/checkpoint");
+
+ /**
+ * Create the same number of Kinesis Receivers/DStreams as stream shards, then union
+ * them all.
+ */
+ var allStreams: DStream[Array[Byte]] = KinesisUtils.createStream(ssc, appName, stream,
+ endpoint, checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the checkpoint interval */
+ allStreams.checkpoint(checkpointInterval)
+ for (i <- 1 until numStreams) {
+ /** Create a new Receiver/DStream for each stream shard */
+ val dStream = KinesisUtils.createStream(ssc, appName, stream, endpoint,
+ checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the Spark checkpoint interval */
+ dStream.checkpoint(checkpointInterval)
+
+ /** Union with the existing streams */
+ allStreams = allStreams.union(dStream)
--- End diff --
The better way to do this Union is to do the followings.
```
val kafkaStreams = (0 until numStream).map { i =>
KinesisUtils.createStream(ssc, ...
}
val unifiedStream = ssc.union(kafkaStreams)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15623324
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param appName Kinesis Application Name. Kinesis Apps are mapped to Kinesis Streams
--- End diff --
Should `@param ssc StreamingContext object`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r18240973
--- Diff: extras/kinesis-asl/src/main/resources/log4j.properties ---
@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+log4j.rootCategory=WARN, console
--- End diff --
question: why do we need this log4j file here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-49132892
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15623527
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param appName Kinesis Application Name. Kinesis Apps are mapped to Kinesis Streams
+ * by the Kinesis Client Library. If you change the App name or Stream name,
+ * the KCL will throw errors.
+ * @param stream Kinesis Stream Name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types
+ * of checkpoints.
+ * @param initialPositionInStream in the absence of Kinesis checkpoint info, this is the
+ * worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours
+ * (InitialPositionInStream.TRIM_HORIZON) or the tip of the stream
+ * (InitialPositionInStream.LATEST).
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk
+ * of processing records more than once.
+ * @param storageLevel The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory
+ * and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ appName: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream,
+ storageLevel: StorageLevel): ReceiverInputDStream[Array[Byte]] = {
+ ssc.receiverStream(new KinesisReceiver(appName, stream, endpoint, checkpointIntervalMillis,
+ initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param appName Kinesis Application Name. Kinesis Apps are mapped to Kinesis Streams
+ * by the Kinesis Client Library. If you change the App name or Stream name,
+ * the KCL will throw errors.
+ * @param stream Kinesis Stream Name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types
+ * of checkpoints.
+ * @param initialPositionInStream in the absence of Kinesis checkpoint info, this is the
+ * worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours
+ * (InitialPositionInStream.TRIM_HORIZON) or the tip of the stream
+ * (InitialPositionInStream.LATEST).
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk
+ * of processing records more than once.
--- End diff --
Same comment as above. No default any more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15623534
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param appName Kinesis Application Name. Kinesis Apps are mapped to Kinesis Streams
+ * by the Kinesis Client Library. If you change the App name or Stream name,
+ * the KCL will throw errors.
+ * @param stream Kinesis Stream Name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types
+ * of checkpoints.
+ * @param initialPositionInStream in the absence of Kinesis checkpoint info, this is the
+ * worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours
+ * (InitialPositionInStream.TRIM_HORIZON) or the tip of the stream
+ * (InitialPositionInStream.LATEST).
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk
+ * of processing records more than once.
+ * @param storageLevel The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory
+ * and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ appName: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream,
+ storageLevel: StorageLevel): ReceiverInputDStream[Array[Byte]] = {
+ ssc.receiverStream(new KinesisReceiver(appName, stream, endpoint, checkpointIntervalMillis,
+ initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
--- End diff --
Same comment as with ssc, should be `@param ssc StreamingContext object`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15651971
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param appName Kinesis Application Name. Kinesis Apps are mapped to Kinesis Streams
+ * by the Kinesis Client Library. If you change the App name or Stream name,
+ * the KCL will throw errors.
+ * @param stream Kinesis Stream Name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15623553
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param appName Kinesis Application Name. Kinesis Apps are mapped to Kinesis Streams
+ * by the Kinesis Client Library. If you change the App name or Stream name,
+ * the KCL will throw errors.
+ * @param stream Kinesis Stream Name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
--- End diff --
Please set the field names correctly. Otherwise the scala docs would not work right.
`@param checkpointIntervalMillis Kinesis checkpoint interval in milliseconds`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50938687
QA tests have started for PR 1434. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17701/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15721137
--- Diff: dev/audit-release/audit_release.py ---
@@ -105,7 +105,7 @@ def get_url(url):
"spark-core", "spark-bagel", "spark-mllib", "spark-streaming", "spark-repl",
"spark-graphx", "spark-streaming-flume", "spark-streaming-kafka",
"spark-streaming-mqtt", "spark-streaming-twitter", "spark-streaming-zeromq",
- "spark-catalyst", "spark-sql", "spark-hive"
+ "spark-catalyst", "spark-sql", "spark-hive", "kinesis-asl"
--- End diff --
I think this needs to change to `spark-streaming-kinesis-asl` since we change the name of the artifact.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15656425
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala ---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.util.List
+
+import scala.collection.JavaConversions.asScalaBuffer
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.Logging
+
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer
+import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownReason
+import com.amazonaws.services.kinesis.model.Record
+
+/**
+ * Kinesis-specific implementation of the Kinesis Client Library (KCL) IRecordProcessor.
+ * This implementation operates on the Array[Byte] from the KinesisReceiver.
+ * The Kinesis Worker creates an instance of this KinesisRecordProcessor upon startup.
+ *
+ * @param Kinesis receiver
+ * @param workerId for logging purposes
+ * @param checkpointState represents the checkpoint state including the next time a
+ * checkpoint is needed. it's injected here for mocking purposes.
--- End diff --
This lines has a tab. Remove.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15616231
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): ReceiverInputDStream[Array[Byte]] = {
+
+ ssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return JavaReceiverInputDStream[Array[Byte]]
+ */
+ def createJavaStream(
+ jssc: JavaStreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): JavaReceiverInputDStream[Array[Byte]] = {
+
+ jssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create checkpoint state using the existing system clock
+ * @param checkpointIntervalMillis
+ */
+ def createCheckpointState(checkpointIntervalMillis: Long): CheckpointState = {
+ new CheckpointState(checkpointIntervalMillis)
--- End diff --
i removed this helper method as it is only used in 1 place.
this is passed in to KinesisRecordProcessor from the KinesisReceiver. it's exposed for testability/mockability as demonstrated in the KinesisReceiverSuite of Scala tests. (ie. lines 81, 83, 86)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15727694
--- Diff: dev/audit-release/sbt_app_kinesis/build.sbt ---
@@ -0,0 +1,30 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+name := "Kinesis Test"
+
+version := "1.0"
+
+scalaVersion := System.getenv.get("SCALA_VERSION")
+
+libraryDependencies += "org.apache.spark" %% "spark-core" % System.getenv.get("SPARK_VERSION")
+libraryDependencies += "org.apache.spark" %% "spark-streaming" % System.getenv.get("SPARK_VERSION")
--- End diff --
gotcha
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15561003
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
--- End diff --
Can you mark this as Experimental API? It is a class annotation, search through spark for "@ExperimentalAPI" to see how its used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15724763
--- Diff: extras/kinesis-asl/src/test/resources/log4j.properties ---
@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Set everything to be logged to the file streaming/target/unit-tests.log
--- End diff --
This path is probably wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15726533
--- Diff: extras/kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.examples.streaming;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.regex.Pattern;
+
+import org.apache.log4j.Logger;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import org.apache.spark.api.java.function.Function2;
+import org.apache.spark.api.java.function.PairFunction;
+import org.apache.spark.storage.StorageLevel;
+import org.apache.spark.streaming.Duration;
+import org.apache.spark.streaming.api.java.JavaDStream;
+import org.apache.spark.streaming.api.java.JavaPairDStream;
+import org.apache.spark.streaming.api.java.JavaStreamingContext;
+import org.apache.spark.streaming.kinesis.KinesisUtils;
+
+import scala.Tuple2;
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.kinesis.AmazonKinesisClient;
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
+import com.google.common.collect.Lists;
+
+/**
+ * Java-friendly Kinesis Spark Streaming WordCount example
+ *
+ * See http://spark.apache.org/docs/latest/streaming-kinesis.html for more details
+ * on the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receiver) per shard
+ * for the given stream.
+ * It then starts pulling from the last checkpointed sequence number of the given
+ * <stream-name> and <endpoint-url>.
+ *
+ * Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials
+ * in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: JavaKinesisWordCountASL <stream-name> <endpoint-url>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service
+ * (ie. https://kinesis.us-east-1.amazonaws.com)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ $SPARK_HOME/bin/run-example \
+ * org.apache.spark.examples.streaming.JavaKinesisWordCountASL mySparkStream \
+ * https://kinesis.us-east-1.amazonaws.com
+ *
+ * There is a companion helper class called KinesisWordCountProducerASL which puts dummy data
+ * onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducerASL are provided in the class definition.
+ */
+public final class JavaKinesisWordCountASL {
+ private static final Pattern WORD_SEPARATOR = Pattern.compile(" ");
+ private static final Logger logger = Logger.getLogger(JavaKinesisWordCountASL.class);
+
+ /*
+ * Make the constructor private to enforce singleton
+ */
+ private JavaKinesisWordCountASL() {
+ }
+
+ public static void main(String[] args) {
+ /*
+ * Check that all required args were passed in.
+ */
+ if (args.length < 2) {
+ System.err.println(
+ "|Usage: KinesisWordCount <stream-name> <endpoint-url>\n" +
+ "| <stream-name> is the name of the Kinesis stream\n" +
+ "| <endpoint-url> is the endpoint of the Kinesis service\n" +
+ "| (e.g. https://kinesis.us-east-1.amazonaws.com)\n");
+ System.exit(1);
+ }
+
+ StreamingExamples.setStreamingLogLevels();
+
+ /* Populate the appropriate variables from the given args */
+ String streamName = args[0];
+ String endpointUrl = args[1];
+ /* Set the batch interval to a fixed 2000 millis (2 seconds) */
+ Duration batchInterval = new Duration(2000);
+
+ /* Create a Kinesis client in order to determine the number of shards for the given stream */
+ AmazonKinesisClient kinesisClient = new AmazonKinesisClient(
+ new DefaultAWSCredentialsProviderChain());
+ kinesisClient.setEndpoint(endpointUrl);
+
+ /* Determine the number of shards from the stream */
+ int numShards = kinesisClient.describeStream(streamName)
+ .getStreamDescription().getShards().size();
+
+ /* In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard */
+ int numStreams = numShards;
+
+ /* Must add 1 more thread than the number of receivers or the output won't show properly from the driver */
+ int numSparkThreads = numStreams + 1;
+
+ /* Setup the Spark config. */
+ SparkConf sparkConfig = new SparkConf().setAppName("KinesisWordCount").setMaster(
+ "local[" + numSparkThreads + "]");
+
+ /* Kinesis checkpoint interval. Same as batchInterval for this example. */
+ Duration checkpointInterval = batchInterval;
+
+ /* Setup the StreamingContext */
+ JavaStreamingContext jssc = new JavaStreamingContext(sparkConfig, batchInterval);
+
+ /* Setup the checkpoint directory used by Spark Streaming */
+ jssc.checkpoint("/tmp/checkpoint");
+
+ /* Create the same number of Kinesis DStreams/Receivers as Kinesis stream's shards */
+ List<JavaDStream<byte[]>> streamsList = new ArrayList<JavaDStream<byte[]>>(numStreams);
+ for (int i = 0; i < numStreams; i++) {
+ streamsList.add(
+ KinesisUtils.createStream(jssc, streamName, endpointUrl, checkpointInterval,
+ InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2())
+ );
+ }
+
+ /* Union all the streams if there is more than 1 stream */
+ JavaDStream<byte[]> unionStreams;
+ if (streamsList.size() > 1) {
+ unionStreams = jssc.union(streamsList.get(0), streamsList.subList(1, streamsList.size()));
+ } else {
+ /* Otherwise, just use the 1 stream */
+ unionStreams = streamsList.get(0);
+ }
+
+ /*
+ * Split each line of the union'd DStreams into multiple words using flatMap to produce the collection.
+ * Convert lines of byte[] to multiple Strings by first converting to String, then splitting on WORD_SEPARATOR.
+ */
+ JavaDStream<String> words = unionStreams.flatMap(new FlatMapFunction<byte[], String>() {
+ @Override
+ public Iterable<String> call(byte[] line) {
+ return Lists.newArrayList(WORD_SEPARATOR.split(new String(line)));
+ }
+ });
+
+ /* Map each word to a (word, 1) tuple, then reduce/aggregate by key. */
+ JavaPairDStream<String, Integer> wordCounts = words.mapToPair(
+ new PairFunction<String, String, Integer>() {
+ @Override
+ public Tuple2<String, Integer> call(String s) {
+ return new Tuple2<String, Integer>(s, 1);
+ }
+ }).reduceByKey(new Function2<Integer, Integer, Integer>() {
+ @Override
+ public Integer call(Integer i1, Integer i2) {
+ return i1 + i2;
+ }
+ });
+
+ /* Print the first 10 wordCounts by key */
--- End diff --
nit: word counts by key doesnt make sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15722833
--- Diff: examples/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming
+
+import java.nio.ByteBuffer
+
+import scala.util.Random
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
+import org.apache.spark.streaming.kinesis.KinesisUtils
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.model.PutRecordRequest
+
+/**
+ * Kinesis Spark Streaming WordCount example.
+ *
+ * See http://spark.apache.org/docs/latest/streaming-kinesis.html for more details on
+ * the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receiver) per shard
+ * for the given stream.
+ * It then starts pulling from the last checkpointed sequence number of the given
+ * <stream-name> and <endpoint-url>.
+ *
+ * Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials
+ * in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: KinesisWordCountASL <stream-name> <endpoint-url>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service
+ * (ie. https://kinesis.us-east-1.amazonaws.com)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ $SPARK_HOME/bin/run-example \
+ * org.apache.spark.examples.streaming.KinesisWordCountASL mySparkStream \
+ * https://kinesis.us-east-1.amazonaws.com
+ *
+ * There is a companion helper class below called KinesisWordCountProducerASL which puts
+ * dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducerASL are provided in that class definition.
+ */
+object KinesisWordCountASL extends Logging {
+ def main(args: Array[String]) {
+/**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 2) {
+ System.err.println(
+ """
+ |Usage: KinesisWordCount <stream-name> <endpoint-url>
+ | <stream-name> is the name of the Kinesis stream
+ | <endpoint-url> is the endpoint of the Kinesis service
+ | (e.g. https://kinesis.us-east-1.amazonaws.com)
+ """.stripMargin)
+ System.exit(1)
+ }
+
+ StreamingExamples.setStreamingLogLevels()
+
+ /** Populate the appropriate variables from the given args */
+ val Array(streamName, endpointUrl) = args
+
+ /** Determine the number of shards from the stream */
+ val kinesisClient = new AmazonKinesisClient(new DefaultAWSCredentialsProviderChain())
+ kinesisClient.setEndpoint(endpointUrl)
+ val numShards = kinesisClient.describeStream(streamName).getStreamDescription().getShards()
+ .size()
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard. */
+ val numStreams = numShards
+
+ /**
+ * numSparkThreads should be 1 more thread than the number of receivers.
+ * This leaves one thread available for actually processing the data.
+ */
+ val numSparkThreads = numStreams + 1
+
+ /** Setup the and SparkConfig and StreamingContext */
+ /** Spark Streaming batch interval */
+ val batchInterval = Milliseconds(2000)
+ val sparkConfig = new SparkConf().setAppName("KinesisWordCount")
+ .setMaster(s"local[$numSparkThreads]")
+ val ssc = new StreamingContext(sparkConfig, batchInterval)
+ /** Setup the checkpoint directory used by Spark Streaming */
+ ssc.checkpoint("/tmp/checkpoint");
--- End diff --
Dont need checkpoint directory in this example.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15670682
--- Diff: dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala ---
@@ -47,7 +47,14 @@ object SimpleApp {
System.exit(-1)
}
if (foundGanglia) {
- println("Ganglia sink was loaded via spark-core")
+ println("Ganglia sink was loaded via spark-ganglia-lgpl")
+ System.exit(-1)
+ }
+
+ // Remove kinesis from default build due to ASL license issue
+ val foundKinesis = Try(Class.forName("org.apache.spark.streaming.kinesis.KinesisUtils")).isSuccess
+ if (foundKinesis) {
+ println("Kinesis was loaded via kinesis-asl")
--- End diff --
This statement should be "Kinesis was loaded via spark-core"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15552831
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+import java.nio.ByteBuffer
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.
+ *
+ * @param app name
+ * @param Kinesis stream name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * @param persistence strategy for RDDs and DStreams.
+ */
+private[streaming] class KinesisReceiver(
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream,
+ storageLevel: StorageLevel)
+ extends Receiver[Array[Byte]](storageLevel) with Logging { receiver =>
+
+ /**
+ * The lazy val's below will get instantiated in the remote Executor after the closure is shipped to the Spark Worker.
+ * These are all lazy because they're from third-party Amazon libraries and are not Serializable.
+ * If they're not marked lazy, they will cause NotSerializableExceptions when they're shipped to the Spark Worker.
+ */
+
+ /**
+ * workerId is lazy because we want the address of the actual Worker where the code runs - not the Driver's ip address.
+ * This makes a difference when running in a cluster.
+ */
+ lazy val workerId = InetAddress.getLocalHost.getHostAddress() + ":" + UUID.randomUUID()
--- End diff --
If you are adding a random UUID to this, do you really need the workerId to be actually the IP of the worker?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15727602
--- Diff: extras/kinesis-asl/pom.xml ---
@@ -0,0 +1,99 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+~ Licensed to the Apache Software Foundation (ASF) under one or more
+~ contributor license agreements. See the NOTICE file distributed with
+~ this work for additional information regarding copyright ownership.
+~ The ASF licenses this file to You under the Apache License, Version 2.0
+~ (the "License"); you may not use this file except in compliance with
+~ the License. You may obtain a copy of the License at
+~
+~ http://www.apache.org/licenses/LICENSE-2.0
+~
+~ Unless required by applicable law or agreed to in writing, software
+~ distributed under the License is distributed on an "AS IS" BASIS,
+~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+~ See the License for the specific language governing permissions and
+~ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-parent</artifactId>
+ <version>1.1.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+
+ <!--
+ Kinesis integration is not included by default due to ASL-licensed code.
+ Note: This project - if activated - is packaged with the main Spark assembly.
--- End diff --
Haha, didnt see your response. Good :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15685887
--- Diff: examples/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCount.scala ---
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming
+
+import java.nio.ByteBuffer
+
+import scala.util.Random
+
+import org.apache.log4j.Level
+import org.apache.log4j.Logger
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkContext.rddToOrderedRDDFunctions
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer
+import org.apache.spark.streaming.kinesis.KinesisUtils
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.model.PutRecordRequest
+
+/**
+ * Kinesis Spark Streaming WordCount example.
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on
+ * the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the
+ * given stream.
+ * It then starts pulling from the last checkpointed sequence number of the given
+ * <stream-name> and <endpoint-url>.
+ *
+ * Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials
+ * in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: KinesisWordCount <stream-name> <endpoint-url>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service
+ * (ie. https://kinesis.us-east-1.amazonaws.com)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ $SPARK_HOME/bin/run-example \
+ * org.apache.spark.examples.streaming.KinesisWordCount mySparkStream \
+ * https://kinesis.us-east-1.amazonaws.com
+ *
+ * There is a companion helper class below called KinesisWordCountProducer which puts
+ * dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in that class definition.
+ */
+object KinesisWordCount extends Logging {
+ val WordSeparator = " "
+
+ def main(args: Array[String]) {
+/**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 2) {
+ System.err.println("Usage: KinesisWordCount <stream-name> <endpoint-url>")
+ System.exit(1)
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency
+ * on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements()
+ if (!log4jInitialized) {
+ /**
+ * We first log something to initialize Spark's default logging,
+ * then we override the logging level.
+ * */
+ logInfo("Setting log level to [INFO] for streaming example." +
+ " To override add a custom log4j.properties to the classpath.")
+
+ Logger.getRootLogger().setLevel(Level.INFO)
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ val Array(stream, endpoint) = args
+ val batchIntervalMillis = 2000
+
+ /** Create a Kinesis client in order to determine the number of shards for the given stream */
+ val kinesisClient = new AmazonKinesisClient(new DefaultAWSCredentialsProviderChain())
+ kinesisClient.setEndpoint(endpoint)
+
+ /** Determine the number of shards from the stream */
+ val numShards = kinesisClient.describeStream(stream).getStreamDescription().getShards().size()
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard.*/
+ val numStreams = numShards
+
+ /**
+ * Must add 1 more thread than the number of receivers or the output won't show properly
+ * from the driver
+ */
+ val numSparkThreads = numStreams + 1
+
+ /** Set the app name */
+ val appName = "KinesisWordCount"
+
+ /** Setup the Spark config. */
+ val sparkConfig = new SparkConf().setAppName(appName).setMaster(s"local[$numSparkThreads]")
+
+ /**
+ * Set the batch interval.
+ * Records will be pulled from the Kinesis stream and stored as a single DStream within Spark
+ * every batch interval.
+ */
+ val batchInterval = Milliseconds(batchIntervalMillis)
+
+ /**
+ * It's recommended that you perform a Spark checkpoint between 5 and 10 times the batch
+ * interval.
+ * While this is the Spark checkpoint interval, we're going to use it for the Kinesis
+ * checkpoint interval, as well.
+ * For example purposes, we'll just use the batchInterval.
+ */
+ val checkpointInterval = batchInterval
+
+ /** Setup the StreamingContext */
+ val ssc = new StreamingContext(sparkConfig, batchInterval)
+
+ /** Setup the checkpoint directory used by Spark Streaming */
+ ssc.checkpoint("/tmp/checkpoint");
+
+ /**
+ * Create the same number of Kinesis Receivers/DStreams as stream shards, then union
+ * them all.
+ */
+ var allStreams: DStream[Array[Byte]] = KinesisUtils.createStream(ssc, appName, stream,
+ endpoint, checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the checkpoint interval */
+ allStreams.checkpoint(checkpointInterval)
+ for (i <- 1 until numStreams) {
+ /** Create a new Receiver/DStream for each stream shard */
+ val dStream = KinesisUtils.createStream(ssc, appName, stream, endpoint,
+ checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the Spark checkpoint interval */
+ dStream.checkpoint(checkpointInterval)
+
+ /** Union with the existing streams */
+ allStreams = allStreams.union(dStream)
--- End diff --
very cool. however, as part of the simplification effort, i remove this altogether. i'm only creating 1 dstream
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15657421
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessorUtils.scala ---
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import scala.util.Random
+
+import org.apache.spark.Logging
+
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+
+
+/**
+ * Helper for the KinesisRecordProcessor.
+ */
+private[kinesis] object KinesisRecordProcessorUtils extends Logging {
--- End diff --
This is a nit pick. But such utils function can often be added in the companion object of the same class. So this could have been added to `object KinesisRecordProcessor`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15568742
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCount.scala ---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming
+
+import java.nio.ByteBuffer
+import org.apache.log4j.Level
+import org.apache.log4j.Logger
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkContext._
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer
+import org.apache.spark.streaming.kinesis.KinesisUtils
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.model.PutRecordRequest
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import org.apache.spark.streaming.dstream.DStream
+
+/**
+ * Kinesis Spark Streaming WordCount example.
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the given stream.
+ * It then starts pulling from the tip of the given <stream-name> and <endpoint-url> at the given <batch-interval>.
+ * Because we're pulling from the tip (InitialPositionInStream.LATEST), only new stream data will be picked up after the KinesisReceiver starts.
+ * This could lead to missed records if data is added to the stream while no KinesisReceivers are running.
+ * In production, you'll want to switch to InitialPositionInStream.TRIM_HORIZON which will read up to 24 hours (Kinesis limit) of previous stream data
+ * depending on the checkpoint frequency.
+ *
+ * InitialPositionInStream.TRIM_HORIZON may lead to duplicate processing of records depending on the checkpoint frequency.
+ * Record processing should be idempotent when possible.
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: KinesisWordCount <stream-name> <endpoint-url> <batch-interval>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service (ie. https://kinesis.us-east-1.amazonaws.com)
+ * <batch-interval> is the batch interval in millis (ie. 1000ms)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ bin/run-kinesis-example \
+ * org.apache.spark.examples.streaming.KinesisWordCount mySparkStream https://kinesis.us-east-1.amazonaws.com 100
+ *
+ * There is a companion helper class below called KinesisWordCountProducer which puts dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in that class definition.
+ */
+object KinesisWordCount extends Logging {
+ val WordSeparator = " "
+
+ def main(args: Array[String]) {
+/**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 3) {
+ System.err.println("Usage: KinesisWordCount <stream-name> <endpoint-url> <batch-interval>")
+ System.exit(1)
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements
+ if (!log4jInitialized) {
+ /** We first log something to initialize Spark's default logging, then we override the logging level. */
+ logInfo("Setting log level to [INFO] for streaming example." +
+ " To override add a custom log4j.properties to the classpath.")
+
+ Logger.getRootLogger().setLevel(Level.INFO)
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ val Array(stream, endpoint, batchIntervalMillisStr) = args
+ val batchIntervalMillis = batchIntervalMillisStr.toInt
+
+ /** Create a Kinesis client in order to determine the number of shards for the given stream */
+ val KinesisClient = new AmazonKinesisClient(new DefaultAWSCredentialsProviderChain());
+
+ /** Determine the number of shards from the stream */
+ val numShards = KinesisClient.describeStream(stream).getStreamDescription().getShards().size()
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStreams for each stream shard */
+ val numStreams = numShards
+
+ /** Must add 1 more thread than the number of receivers or the output won't show properly from the driver */
+ val numSparkThreads = numStreams + 1
+
+ /** Set the app name */
+ val app = "KinesisWordCount"
+
+ /** Setup the Spark config. */
+ val sparkConfig = new SparkConf().setAppName(app).setMaster(s"local[$numSparkThreads]")
+
+ /**
+ * Set the batch interval.
+ * Records will be pulled from the Kinesis stream and stored as a single DStream within Spark every batch interval.
+ */
+ val batchInterval = Milliseconds(batchIntervalMillis)
+
+ /**
+ * It's recommended that you perform a Spark checkpoint between 5 and 10 times the batch interval.
+ * While this is the Spark checkpoint interval, we're going to use it for the Kinesis checkpoint interval, as well.
+ */
+ val checkpointInterval = batchInterval * 5
+
+ /** Setup the StreamingContext */
+ val ssc = new StreamingContext(sparkConfig, batchInterval)
+
+ /** Setup the checkpoint directory used by Spark Streaming */
+ ssc.checkpoint("/tmp/checkpoint");
+
+ /** Create the same number of Kinesis Receivers/DStreams as stream shards, then union them all */
+ var allStreams: DStream[Array[Byte]] = KinesisUtils.createStream(ssc, app, stream, endpoint, checkpointInterval.milliseconds,
+ InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2)
+ /** Set the checkpoint interval */
+ allStreams.checkpoint(checkpointInterval)
+ for (i <- 1 until numStreams) {
+ /** Create a new Receiver/DStream for each stream shard */
+ val dStream = KinesisUtils.createStream(ssc, app, stream, endpoint, checkpointInterval.milliseconds,
+ InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2)
+ /** Set the Spark checkpoint interval */
+ dStream.checkpoint(checkpointInterval)
+
+ /** Union with the existing streams */
+ allStreams = allStreams.union(dStream)
+ }
+
+ /** This implementation uses the String-based KinesisRecordSerializer impl */
+ val recordSerializer = new KinesisStringRecordSerializer()
+
+ /**
+ * Sort and print the given dstream.
+ * This is an Output Operation that will materialize the underlying DStream.
+ * Everything up to this point is a lazy Transformation Operation.
+ *
+ * @param description of the dstream for logging purposes
+ * @param dstream to sort and print
+ */
+ def sortAndPrint(description: String, dstream: DStream[(String,Int)]) = {
+ dstream.foreachRDD((batch, endOfWindowTime) => {
--- End diff --
This is great example with a lot of functionality but for a newbie trying to learn Spark Streaming + Kinesis, this example looks a little scary because of its size. All we want to demonstrate in this example is how to use KinesisUtils, not that you can use window operations and stuff. There are other examples for that. So I would prefer this example being very short and to the point, like KafkaWordCount. Can you please change it to something like that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15556089
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+import java.nio.ByteBuffer
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.
+ *
+ * @param app name
+ * @param Kinesis stream name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * @param persistence strategy for RDDs and DStreams.
+ */
+private[streaming] class KinesisReceiver(
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream,
+ storageLevel: StorageLevel)
+ extends Receiver[Array[Byte]](storageLevel) with Logging { receiver =>
+
+ /**
+ * The lazy val's below will get instantiated in the remote Executor after the closure is shipped to the Spark Worker.
+ * These are all lazy because they're from third-party Amazon libraries and are not Serializable.
+ * If they're not marked lazy, they will cause NotSerializableExceptions when they're shipped to the Spark Worker.
+ */
+
+ /**
+ * workerId is lazy because we want the address of the actual Worker where the code runs - not the Driver's ip address.
+ * This makes a difference when running in a cluster.
+ */
+ lazy val workerId = InetAddress.getLocalHost.getHostAddress() + ":" + UUID.randomUUID()
--- End diff --
Aah, that is a good point! Lets keep it then!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15657271
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisStringRecordSerializer.scala ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+
+/**
+ * Implementation of KinesisRecordSerializer to convert Array[Byte] to/from String.
+ */
+class KinesisStringRecordSerializer extends KinesisRecordSerializer[String] with Logging {
--- End diff --
Is this class supposed to be publcly visible? If not, do the expected :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15656394
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+
+import com.amazonaws.auth.AWSCredentialsProvider
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver)
+ * as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers
+ * to run within a Spark Executor.
+ *
+ * @param appName Kinesis Application Name. Kinesis apps are mapped to Kinesis streams
+ * by the Kinesis Client Library. If you change the app name or stream name,
+ * the KCL will throw errors.
+ * @param stream Kinesis stream name
+ * @param endpoint url of Kinesis service
+ * @param checkpointIntervalMillis for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types
+ * of checkpoints.
+ * @param initialPositionInStream in the absence of Kinesis checkpoint info, this is the worker's initial
--- End diff --
This lines is more than 100 chars
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15616323
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+import java.nio.ByteBuffer
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.
+ *
+ * @param app name
+ * @param Kinesis stream name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * @param persistence strategy for RDDs and DStreams.
+ */
+private[streaming] class KinesisReceiver(
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream,
+ storageLevel: StorageLevel)
+ extends Receiver[Array[Byte]](storageLevel) with Logging { receiver =>
+
+ /**
+ * The lazy val's below will get instantiated in the remote Executor after the closure is shipped to the Spark Worker.
+ * These are all lazy because they're from third-party Amazon libraries and are not Serializable.
+ * If they're not marked lazy, they will cause NotSerializableExceptions when they're shipped to the Spark Worker.
+ */
+
+ /**
+ * workerId is lazy because we want the address of the actual Worker where the code runs - not the Driver's ip address.
+ * This makes a difference when running in a cluster.
+ */
+ lazy val workerId = InetAddress.getLocalHost.getHostAddress() + ":" + UUID.randomUUID()
+
+ /**
+ * This impl uses the DefaultAWSCredentialsProviderChain per the following url:
+ * http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html
+ * and searches for credentials in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
+ * Instance profile credentials delivered through the Amazon EC2 metadata service
+ */
+ lazy val credentialsProvider = new DefaultAWSCredentialsProviderChain()
+
+ /** Create a KCL config instance. */
+ lazy val KinesisClientLibConfiguration = new KinesisClientLibConfiguration(app, stream, credentialsProvider, workerId)
+ .withKinesisEndpoint(endpoint).withInitialPositionInStream(initialPositionInStream).withTaskBackoffTimeMillis(500)
--- End diff --
ah, i just looked at FlumeReceiver. i see the workaround. lemme code it up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15727693
--- Diff: dev/audit-release/audit_release.py ---
@@ -105,7 +105,7 @@ def get_url(url):
"spark-core", "spark-bagel", "spark-mllib", "spark-streaming", "spark-repl",
"spark-graphx", "spark-streaming-flume", "spark-streaming-kafka",
"spark-streaming-mqtt", "spark-streaming-twitter", "spark-streaming-zeromq",
- "spark-catalyst", "spark-sql", "spark-hive"
+ "spark-catalyst", "spark-sql", "spark-hive", "kinesis-asl"
--- End diff --
changed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15564390
--- Diff: extras/spark-kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala ---
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.nio.ByteBuffer
+import java.nio.CharBuffer
+import java.nio.charset.Charset
+import scala.collection.JavaConversions.seqAsJavaList
+import org.scalatest.BeforeAndAfter
+import org.scalatest.FunSuite
+import org.scalatest.Matchers
+import org.scalatest.PrivateMethodTester
+import org.scalatest.mock.EasyMockSugar
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer
+import com.amazonaws.services.kinesis.model.Record
--- End diff --
Please sort import based on the coding style guide
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15685828
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisStringRecordSerializer.scala ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+
+/**
+ * Implementation of KinesisRecordSerializer to convert Array[Byte] to/from String.
+ */
+class KinesisStringRecordSerializer extends KinesisRecordSerializer[String] with Logging {
--- End diff --
i removed the Serializer abstraction and am just using basic byte[] <-> String conversions
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50443974
@pdeyhim can you take a look over this too when you have a chance?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15549518
--- Diff: extras/spark-kinesis-asl/pom.xml ---
@@ -0,0 +1,98 @@
+<?xml version="1.0" encoding="UTF-8"?>
--- End diff --
Aah, I understand, you must have followed the existing `spark-ganglia-gpl` directory name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15615786
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+import java.nio.ByteBuffer
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.
+ *
+ * @param app name
+ * @param Kinesis stream name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * @param persistence strategy for RDDs and DStreams.
+ */
+private[streaming] class KinesisReceiver(
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream,
+ storageLevel: StorageLevel)
+ extends Receiver[Array[Byte]](storageLevel) with Logging { receiver =>
+
+ /**
+ * The lazy val's below will get instantiated in the remote Executor after the closure is shipped to the Spark Worker.
+ * These are all lazy because they're from third-party Amazon libraries and are not Serializable.
+ * If they're not marked lazy, they will cause NotSerializableExceptions when they're shipped to the Spark Worker.
+ */
+
+ /**
+ * workerId is lazy because we want the address of the actual Worker where the code runs - not the Driver's ip address.
+ * This makes a difference when running in a cluster.
+ */
+ lazy val workerId = InetAddress.getLocalHost.getHostAddress() + ":" + UUID.randomUUID()
+
+ /**
+ * This impl uses the DefaultAWSCredentialsProviderChain per the following url:
+ * http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html
+ * and searches for credentials in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
+ * Instance profile credentials delivered through the Amazon EC2 metadata service
+ */
+ lazy val credentialsProvider = new DefaultAWSCredentialsProviderChain()
+
+ /** Create a KCL config instance. */
+ lazy val KinesisClientLibConfiguration = new KinesisClientLibConfiguration(app, stream, credentialsProvider, workerId)
+ .withKinesisEndpoint(endpoint).withInitialPositionInStream(initialPositionInStream).withTaskBackoffTimeMillis(500)
--- End diff --
yeah, this is a little awkward. the problem is that these Kinesis classes are not Serializable, so when they get shipped to the Spark Worker, i get the dreaded NotSerializableException.
this is why they need to be lazy. if i create them each time onStart() is called, i'll lose the laziness.
any suggestions for a workaround?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50525716
QA results for PR 1434:<br>- This patch PASSES unit tests.<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17365/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15617760
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15617773
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15726526
--- Diff: extras/kinesis-asl/pom.xml ---
@@ -0,0 +1,99 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+~ Licensed to the Apache Software Foundation (ASF) under one or more
+~ contributor license agreements. See the NOTICE file distributed with
+~ this work for additional information regarding copyright ownership.
+~ The ASF licenses this file to You under the Apache License, Version 2.0
+~ (the "License"); you may not use this file except in compliance with
+~ the License. You may obtain a copy of the License at
+~
+~ http://www.apache.org/licenses/LICENSE-2.0
+~
+~ Unless required by applicable law or agreed to in writing, software
+~ distributed under the License is distributed on an "AS IS" BASIS,
+~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+~ See the License for the specific language governing permissions and
+~ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-parent</artifactId>
+ <version>1.1.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+
+ <!--
+ Kinesis integration is not included by default due to ASL-licensed code.
+ Note: This project - if activated - is packaged with the main Spark assembly.
--- End diff --
Nit: Change this, this will be super confusing in future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15723363
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointState.scala ---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.Duration
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * This is a helper class for managing checkpoint clocks.
+ *
+ * @param checkpointInterval
+ * @param currentClock. Default to current SystemClock if none is passed in (mocking purposes)
+ */
+private[kinesis] class KinesisCheckpointState(
--- End diff --
Good you changed the name!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15623547
--- Diff: project/SparkBuild.scala ---
@@ -62,7 +62,7 @@ object SparkBuild extends PomBuild {
var isAlphaYarn = false
var profiles: mutable.Seq[String] = mutable.Seq.empty
if (Properties.envOrNone("SPARK_GANGLIA_LGPL").isDefined) {
- println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pganglia-lgpl flag.")
+ println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pspark-ganglia-lgpl flag.")
--- End diff --
As per Patrick's point, we probably don't need to change the name of this profile, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15564377
--- Diff: extras/spark-kinesis-asl/src/test/resources/log4j.properties ---
@@ -0,0 +1,42 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Set everything to be logged to the file streaming/target/unit-tests.log
+log4j.rootCategory=WARN, console
+
--- End diff --
Please match this log4j.properties level with that in external/flume/src/test/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15685865
--- Diff: examples/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCount.scala ---
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming
+
+import java.nio.ByteBuffer
+
+import scala.util.Random
+
+import org.apache.log4j.Level
+import org.apache.log4j.Logger
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkContext.rddToOrderedRDDFunctions
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer
+import org.apache.spark.streaming.kinesis.KinesisUtils
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.model.PutRecordRequest
+
+/**
+ * Kinesis Spark Streaming WordCount example.
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on
+ * the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the
+ * given stream.
+ * It then starts pulling from the last checkpointed sequence number of the given
+ * <stream-name> and <endpoint-url>.
+ *
+ * Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials
+ * in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: KinesisWordCount <stream-name> <endpoint-url>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service
+ * (ie. https://kinesis.us-east-1.amazonaws.com)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ $SPARK_HOME/bin/run-example \
+ * org.apache.spark.examples.streaming.KinesisWordCount mySparkStream \
+ * https://kinesis.us-east-1.amazonaws.com
+ *
+ * There is a companion helper class below called KinesisWordCountProducer which puts
+ * dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in that class definition.
+ */
+object KinesisWordCount extends Logging {
+ val WordSeparator = " "
+
+ def main(args: Array[String]) {
+/**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 2) {
+ System.err.println("Usage: KinesisWordCount <stream-name> <endpoint-url>")
+ System.exit(1)
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency
+ * on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements()
+ if (!log4jInitialized) {
+ /**
+ * We first log something to initialize Spark's default logging,
+ * then we override the logging level.
+ * */
+ logInfo("Setting log level to [INFO] for streaming example." +
+ " To override add a custom log4j.properties to the classpath.")
+
+ Logger.getRootLogger().setLevel(Level.INFO)
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ val Array(stream, endpoint) = args
+ val batchIntervalMillis = 2000
+
+ /** Create a Kinesis client in order to determine the number of shards for the given stream */
+ val kinesisClient = new AmazonKinesisClient(new DefaultAWSCredentialsProviderChain())
+ kinesisClient.setEndpoint(endpoint)
+
+ /** Determine the number of shards from the stream */
+ val numShards = kinesisClient.describeStream(stream).getStreamDescription().getShards().size()
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard.*/
+ val numStreams = numShards
+
+ /**
+ * Must add 1 more thread than the number of receivers or the output won't show properly
+ * from the driver
+ */
+ val numSparkThreads = numStreams + 1
+
+ /** Set the app name */
+ val appName = "KinesisWordCount"
+
+ /** Setup the Spark config. */
+ val sparkConfig = new SparkConf().setAppName(appName).setMaster(s"local[$numSparkThreads]")
+
+ /**
+ * Set the batch interval.
+ * Records will be pulled from the Kinesis stream and stored as a single DStream within Spark
+ * every batch interval.
+ */
+ val batchInterval = Milliseconds(batchIntervalMillis)
+
+ /**
+ * It's recommended that you perform a Spark checkpoint between 5 and 10 times the batch
+ * interval.
+ * While this is the Spark checkpoint interval, we're going to use it for the Kinesis
+ * checkpoint interval, as well.
+ * For example purposes, we'll just use the batchInterval.
+ */
+ val checkpointInterval = batchInterval
+
+ /** Setup the StreamingContext */
+ val ssc = new StreamingContext(sparkConfig, batchInterval)
+
+ /** Setup the checkpoint directory used by Spark Streaming */
+ ssc.checkpoint("/tmp/checkpoint");
+
+ /**
+ * Create the same number of Kinesis Receivers/DStreams as stream shards, then union
+ * them all.
+ */
+ var allStreams: DStream[Array[Byte]] = KinesisUtils.createStream(ssc, appName, stream,
+ endpoint, checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the checkpoint interval */
+ allStreams.checkpoint(checkpointInterval)
+ for (i <- 1 until numStreams) {
+ /** Create a new Receiver/DStream for each stream shard */
+ val dStream = KinesisUtils.createStream(ssc, appName, stream, endpoint,
+ checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the Spark checkpoint interval */
+ dStream.checkpoint(checkpointInterval)
--- End diff --
i removed the call to dstream.checkpoint()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15549163
--- Diff: extras/spark-kinesis-asl/pom.xml ---
@@ -0,0 +1,98 @@
+<?xml version="1.0" encoding="UTF-8"?>
--- End diff --
Can you rename is the directory `spark-kinesis-asl` to just `kinesis`. Its already in spark code base, so does not quite need another `spark` in the directory name. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15554012
--- Diff: extras/spark-kinesis-asl/pom.xml ---
@@ -0,0 +1,98 @@
+<?xml version="1.0" encoding="UTF-8"?>
--- End diff --
Okay, I spoke to @pwendell and there is not need to have `spark-` as part of the name directory name and profile name. So can you make it `kinesis-asl` at all places.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by srosenthal <gi...@git.apache.org>.
Github user srosenthal commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15137079
--- Diff: docs/streaming-programming-guide.md ---
@@ -467,6 +468,62 @@ For more details on these additional sources, see the corresponding [API documen
Furthermore, you can also implement your own custom receiver for your sources. See the
[Custom Receiver Guide](streaming-custom-receivers.html).
+### Kinesis
+Build notes:
+<li>Spark supports a Kinesis Streaming Receiver which is not included in the default build due to licensing restrictions.</li>
+<li>_**Note that by embedding this library you will include [ASL](https://aws.amazon.com/asl/)-licensed code in your Spark package**_.</li>
+<li>For sbt users, set the `SPARK_KINESIS_ASL` environment variable before building.</li>
+<li>For Maven users, enable the `-Pspark-kinesis-asl` profile.</li>
+<li>User applications will need to link to the `spark-kinesis-asl` artifact.</li>
+<li>The Spark Kinesis Streaming Receiver source code, examples, tests, and artifacts live in $SPARK_HOME/extras/spark-kinesis-asl.</li>
+
+Deployment and runtime notes:
+<li>Each shard of a stream is processed by one or more KinesisReceiver's managed by the Kinesis Client Library (KCL) Worker.</li>
+<li>Said differently, a single KinesisReceiver can process many shards of a stream.</li>
+<li>You never need more KinesisReceivers than the number of shards in your stream.</li>
+<li>The Kinesis assembly jar must also be present on all worker nodes, as they will need access to the Kinesis Client Library.</li>
+<li>/tmp/checkpoint is a valid and accessible directory on all workers (or locally if running in local mode)</li>
+<li>This code uses the DefaultAWSCredentialsProviderChain and searches for credentials in the following order of precedence:<br/>
+ 1) Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY<br/>
+ 2) Java System Properties - aws.accessKeyId and aws.secretKey<br/>
+ 3) Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs<br/>
+ 4) Instance profile credentials - delivered through the Amazon EC2 metadata service<br/>
+</li>
+<li>You need to setup a Kinesis stream with 1 or more shards per the following:<br/>
+ http://docs.aws.amazon.com/kinesis/latest/dev/step-one-create-stream.html</li>
+<li>When you first start up the KinesisReceiver, the Kinesis Client Library (KCL) needs ~30s to establish connectivity with the AWS Kinesis service,
+retrieve any checkpoint data, and negotiate with other KCL's reading from the same stream.</li>
+<li>During testing, I noticed varying degrees of delays while retrieving records from Kinesis depending on which coffee shop in San Francisco I was working.
+The input and output data eventually matched, but sometimes after an unusually long time.</li>
+<li>Be careful when changing the app name. Kinesis maintains a mapping table in DynamoDB based on this app name (http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app.html#kinesis-record-processor-initialization).
+Changing the app name could lead to Kinesis errors as only 1 logical application can process a stream.</li>
+
+Failure recovery notes:
+<li>The combination of Spark Streaming and Kinesis creates 3 different checkpoints as follows:<br/>
+ 1) RDD data checkpoint (Spark Streaming) - frequency is configurable with DStream.checkpoint(Duration)<br/>
+ 2) RDD metadata checkpoint (Spark Streaming) - frequency is every DStream batch<br/>
+ 3) Kinesis checkpointing (Kinesis) - frequency is controlled by the developer calling ICheckpointer.checkpoint() directly<br/>
+</li>
+<li>During testing, if you see the same data being read from the stream twice, it's likely due to the Kinesis checkpoints not being written.</li>
+<li>Checkpointing too freqently will cause excess load on the AWS checkpoint storage layer and may lead to AWS throttling</li>
+<li>Upon startup, a KinesisReceiver will begin processing records with sequence numbers greater than the last checkpoint sequence number recorded per shard.</li>
+<li>If no checkpoint info exists, the worker will start either from the oldest record available (InitialPositionInStream.TRIM_HORIZON)
+or from the tip/latest (InitialPostitionInStream.LATEST). This is configurable.</li>
+<li>When pulling from the stream tip (InitialPositionInStream.LATEST), only new stream data will be picked up after the KinesisReceiver starts.</li>
+<li>InitialPositionInStream.LATEST could lead to missed records if data is added to the stream while no KinesisReceivers are running.</li>
+<li>In production, you'll want to switch to InitialPositionInStream.TRIM_HORIZON which will read up to 24 hours (Kinesis limit) of previous stream data
+depending on the checkpoint frequency.</li>
+<li>InitialPositionInStream.TRIM_HORIZON may lead to duplicate processing of records depending on the checkpoint frequency.</li>
+<li>Record processing should be idempotent when possible.</li>
+<li>Failed or latent KinesisReceivers will be detected and automatically shutdown/load-balanced by the KCL.</li>
+<li>If possible, explicitly shutdown the worker if a failure occurs.</li>
+
+Example KinesisWordCount (and JavaKiensisWordCount) notes:
--- End diff --
Typo - JavaKinesisWordCount
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15727578
--- Diff: bin/run-example ---
@@ -29,7 +29,9 @@ if [ -n "$1" ]; then
else
echo "Usage: ./bin/run-example <example-class> [example-args]" 1>&2
echo " - set MASTER=XX to use a specific master" 1>&2
- echo " - can use abbreviated example class name (e.g. SparkPi, mllib.LinearRegression)" 1>&2
+ echo " - can use abbreviated example class name relative to com.apache.spark.examples" 1>&2
+ echo " (e.g. SparkPi, mllib.LinearRegression, streaming.KinesisWordCountASL)" 1>&2
+ echo " - to run the Kinesis Spark Streaming example, make sure you build with -Pkinesis-asl" 1>&2
--- End diff --
makes sense. i'll make sure this is addressed in streaming-kinesis.md.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15723335
--- Diff: extras/kinesis-asl/src/main/resources/log4j.properties ---
@@ -0,0 +1,42 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Set everything to be logged to the file streaming/target/unit-tests.log
+log4j.rootCategory=WARN, console
+
+# File appender
+log4j.appender.file=org.apache.log4j.FileAppender
+log4j.appender.file.append=false
+log4j.appender.file.file=target/unit-tests.log
+log4j.appender.file.layout=org.apache.log4j.PatternLayout
+log4j.appender.file.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss.SSS} %p %c{1}: %m%n
+
+# Console appender
+log4j.appender.console=org.apache.log4j.ConsoleAppender
+log4j.appender.console.target=System.out
+log4j.appender.console.layout=org.apache.log4j.PatternLayout
+log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+
+# Settings to quiet third party logs that are too verbose
+log4j.logger.org.eclipse.jetty=WARN
+log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
+log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
+log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
+
+# Log all Kinesis Streaming messages
+log4j.logger.org.apache.spark.examples.streaming=DEBUG
--- End diff --
These lines were removed in the PR I made to your branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50973644
Thanks you very much @cfregly! I have merged this!!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50701651
QA tests have started for PR 1434. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17530/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15561415
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): ReceiverInputDStream[Array[Byte]] = {
+
+ ssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return JavaReceiverInputDStream[Array[Byte]]
+ */
+ def createJavaStream(
--- End diff --
Please follow the convention of other XYZUtils, where the Java ones are named as createStream as well. Keeps things consistent across Scala and Java.
Also, please add Java unit test to make sure that this function, along with its variations of default params, can be called from Java.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15722812
--- Diff: examples/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.examples.streaming;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.regex.Pattern;
+
+import org.apache.log4j.Logger;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import org.apache.spark.api.java.function.Function2;
+import org.apache.spark.api.java.function.PairFunction;
+import org.apache.spark.storage.StorageLevel;
+import org.apache.spark.streaming.Duration;
+import org.apache.spark.streaming.api.java.JavaDStream;
+import org.apache.spark.streaming.api.java.JavaPairDStream;
+import org.apache.spark.streaming.api.java.JavaStreamingContext;
+import org.apache.spark.streaming.kinesis.KinesisUtils;
+
+import scala.Tuple2;
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.kinesis.AmazonKinesisClient;
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
+import com.google.common.collect.Lists;
+
+/**
+ * Java-friendly Kinesis Spark Streaming WordCount example
+ *
+ * See http://spark.apache.org/docs/latest/streaming-kinesis.html for more details
+ * on the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receiver) per shard
+ * for the given stream.
+ * It then starts pulling from the last checkpointed sequence number of the given
+ * <stream-name> and <endpoint-url>.
+ *
+ * Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials
+ * in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: JavaKinesisWordCountASL <stream-name> <endpoint-url>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service
+ * (ie. https://kinesis.us-east-1.amazonaws.com)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ $SPARK_HOME/bin/run-example \
+ * org.apache.spark.examples.streaming.JavaKinesisWordCountASL mySparkStream \
+ * https://kinesis.us-east-1.amazonaws.com
+ *
+ * There is a companion helper class called KinesisWordCountProducerASL which puts dummy data
+ * onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducerASL are provided in the class definition.
+ */
+public final class JavaKinesisWordCountASL {
+ private static final Pattern WORD_SEPARATOR = Pattern.compile(" ");
+ private static final Logger logger = Logger.getLogger(JavaKinesisWordCountASL.class);
+
+ /**
+ * Make the constructor private to enforce singleton
+ */
+ private JavaKinesisWordCountASL() {
+ }
+
+ public static void main(String[] args) {
+ /**
--- End diff --
Inline comment style should /* ... */ .... /** is reserved only for scala docs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15504376
--- Diff: bin/run-kinesis-example.cmd ---
@@ -0,0 +1,90 @@
+@echo off
--- End diff --
Same comment as the above script.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15618463
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): ReceiverInputDStream[Array[Byte]] = {
+
+ ssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return JavaReceiverInputDStream[Array[Byte]]
+ */
+ def createJavaStream(
--- End diff --
created the junit test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15561926
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): ReceiverInputDStream[Array[Byte]] = {
+
+ ssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return JavaReceiverInputDStream[Array[Byte]]
+ */
+ def createJavaStream(
+ jssc: JavaStreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): JavaReceiverInputDStream[Array[Byte]] = {
+
+ jssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create checkpoint state using the existing system clock
+ * @param checkpointIntervalMillis
+ */
+ def createCheckpointState(checkpointIntervalMillis: Long): CheckpointState = {
+ new CheckpointState(checkpointIntervalMillis)
+ }
+
+ /**
+ * Retry the given amount of times with a random backoff time (millis) less than the given maxBackOffMillis
+ *
+ * @param expression expression to evalute
+ * @param numRetriesLeft number of retries left
+ * @param maxBackOffMillis: max millis between retries
+ *
+ * @return Evaluation of the given expression
+ * @throws Unretryable exception, unexpected exception,
+ * or any exception that persists after numRetriesLeft reaches 0
+ */
+ @annotation.tailrec
+ def retry[T](expression: => T, numRetriesLeft: Int, maxBackOffMillis: Int): T = {
--- End diff --
What is the point of this function? Seems like this is only used in unit-tests? In which case shouldnt this be in the test classes only? Either way, if the user is not expected to use this function directly, this is should not be exposed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15617745
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+import java.nio.ByteBuffer
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.
+ *
+ * @param app name
--- End diff --
updated
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15668098
--- Diff: examples/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCount.java ---
@@ -0,0 +1,294 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.examples.streaming;
+
+import java.util.List;
+import java.util.regex.Pattern;
+
+import org.apache.log4j.Level;
+import org.apache.log4j.Logger;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.api.java.function.Function2;
+import org.apache.spark.api.java.function.PairFunction;
+import org.apache.spark.storage.StorageLevel;
+import org.apache.spark.streaming.Duration;
+import org.apache.spark.streaming.Milliseconds;
+import org.apache.spark.streaming.api.java.JavaDStream;
+import org.apache.spark.streaming.api.java.JavaPairDStream;
+import org.apache.spark.streaming.api.java.JavaStreamingContext;
+import org.apache.spark.streaming.kinesis.KinesisRecordSerializer;
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer;
+import org.apache.spark.streaming.kinesis.KinesisUtils;
+
+import scala.Tuple2;
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.kinesis.AmazonKinesisClient;
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
+import com.google.common.base.Optional;
+import com.google.common.collect.Lists;
+
+/**
+ * Java-friendly Kinesis Spark Streaming WordCount example
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details
+ * on the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard
+ * of the given stream.
+ * It then starts pulling from the last checkpointed sequence number of the given
+ * <stream-name> and <endpoint-url>.
+ *
+ * Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials
+ * in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: JavaKinesisWordCount <stream-name> <endpoint-url>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service
+ * (ie. https://kinesis.us-east-1.amazonaws.com)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ $SPARK_HOME/bin/run-example \
+ * org.apache.spark.examples.streaming.JavaKinesisWordCount mySparkStream \
+ * https://kinesis.us-east-1.amazonaws.com
+ *
+ * There is a companion helper class called KinesisWordCountProducer which puts dummy data
+ * onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in the class definition.
+ */
+public final class JavaKinesisWordCount {
+ private static final Pattern WORD_SEPARATOR = Pattern.compile(" ");
+ private static final Logger logger = Logger.getLogger(JavaKinesisWordCount.class);
+
+ /**
+ * Make the constructor private to enforce singleton
+ */
+ private JavaKinesisWordCount() {
+ }
+
+ public static void main(String[] args) {
+ /**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 2) {
+ System.err.println("Usage: JavaKinesisWordCount <stream-name> <kinesis-endpoint-url>");
+ System.exit(1);
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ boolean log4jInitialized = Logger.getRootLogger().getAllAppenders()
+ .hasMoreElements();
+ if (!log4jInitialized) {
+ /** We first log something to initialize Spark's default logging, then we override the logging level. */
+ Logger.getRootLogger()
+ .info("Setting log level to [ERROR] for streaming example."
+ + " To override add a custom log4j.properties to the classpath.");
+ Logger.getRootLogger().setLevel(Level.ERROR);
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ String stream = args[0];
+ String endpoint = args[1];
+ /** Set the batch interval to a fixed 2000 millis (2 seconds) */
+ Integer batchIntervalMillis = 2000;
+
+ /** Create a Kinesis client in order to determine the number of shards for the given stream */
+ AmazonKinesisClient kinesisClient = new AmazonKinesisClient(
+ new DefaultAWSCredentialsProviderChain());
+ kinesisClient.setEndpoint(endpoint);
+
+ /** Determine the number of shards from the stream */
+ int numShards = kinesisClient.describeStream(stream)
+ .getStreamDescription().getShards().size();
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard */
+ int numStreams = numShards;
+
+ /** Must add 1 more thread than the number of receivers or the output won't show properly from the driver */
+ int numSparkThreads = numStreams + 1;
+
+ /** Set the app name */
+ String appName = "KinesisWordCount";
+
+ /** Setup the Spark config. */
+ SparkConf sparkConfig = new SparkConf().setAppName(appName).setMaster(
+ "local[" + numSparkThreads + "]");
+
+ /**
+ * Set the batch interval.
+ * Records will be pulled from the Kinesis stream and stored as a single DStream within Spark every batch interval.
+ */
+ Duration batchInterval = Milliseconds.apply(batchIntervalMillis);
+
+ /**
+ * It's recommended that you perform a Spark checkpoint between 5 and 10 times the batch interval.
+ * While this is the Spark checkpoint interval, we're going to use it for the Kinesis checkpoint interval, as well.
+ * For example purposes, we'll just use the batchInterval.
+ */
+ Duration checkpointInterval = Milliseconds.apply(batchIntervalMillis);
+
+ /** Setup the StreamingContext */
+ JavaStreamingContext jssc = new JavaStreamingContext(sparkConfig, batchInterval);
+
+ /** Setup the checkpoint directory used by Spark Streaming */
+ jssc.checkpoint("/tmp/checkpoint");
+
+ /** Create the same number of Kinesis Receivers/DStreams as stream shards, then union them all */
+ JavaDStream<byte[]> allStreams = KinesisUtils
+ .createStream(jssc, appName, stream, endpoint, checkpointInterval.milliseconds(),
+ InitialPositionInStream.LATEST);
+ /** Set the checkpoint interval */
+ allStreams.checkpoint(checkpointInterval);
+ for (int i = 1; i < numStreams; i++) {
+ /** Create a new Receiver/DStream for each stream shard */
+ JavaDStream<byte[]> dStream = KinesisUtils
+ .createStream(jssc, appName, stream, endpoint, checkpointInterval.milliseconds(),
+ InitialPositionInStream.LATEST);
+ /** Set the Spark checkpoint interval */
+ dStream.checkpoint(checkpointInterval);
+
+ /** Union with the existing streams */
+ allStreams = allStreams.union(dStream);
+ }
+
+ /** This implementation uses the String-based KinesisRecordSerializer impl */
+ final KinesisRecordSerializer<String> recordSerializer = new KinesisStringRecordSerializer();
+
+ /**
+ * Split each line of the union'd DStreams into multiple words using flatMap to produce the collection.
+ * Convert lines of byte[] to multiple Strings by first converting to String, then splitting on WORD_SEPARATOR
+ * We're caching the result here so that we can use it later without having to re-materialize the underlying RDDs.
+ */
+ JavaDStream<String> words = allStreams.flatMap(new FlatMapFunction<byte[], String>() {
+ /**
+ * Convert lines of byte[] to multiple words split by WORD_SEPARATOR
+ * @param byte array
+ * @return iterable of words split by WORD_SEPARATOR
+ */
+ @Override
+ public Iterable<String> call(byte[] line) {
+ return Lists.newArrayList(WORD_SEPARATOR.split(recordSerializer.deserialize(line)));
+ }
+ }).cache();
+
+ /**
+ * Map each word to a (word, 1) tuple so we can reduce/aggregate later.
+ * We're caching the result here so that we can use it later without having
+ * to re-materialize the underlying RDDs.
+ */
+ JavaPairDStream<String, Integer> wordCounts = words.mapToPair(
+ new PairFunction<String, String, Integer>() {
+ /**
+ * Create the (word, 1) tuple
+ * @param word
+ * @return (word, 1) tuple
+ */
+ @Override
+ public Tuple2<String, Integer> call(String s) {
+ return new Tuple2<String, Integer>(s, 1);
+ }
+ });
+
+ /**
+ * Reduce/aggregate by key
+ * We're caching the result here so that we can use it later without having
+ * to re-materialize the underlying RDDs.
+ */
+ JavaPairDStream<String, Integer> wordCountsByKey = wordCounts.reduceByKey(
+ new Function2<Integer, Integer, Integer>() {
+ @Override
+ public Integer call(Integer i1, Integer i2) {
+ return i1 + i2;
+ }
+ }).cache();
+
+ /** Update the running totals of words. */
--- End diff --
Same comment as the other example. Needs to be simplified.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50792565
QA results for PR 1434:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>public final class JavaKinesisWordCount {<br>class KinesisStringRecordSerializer extends KinesisRecordSerializer[String] with Logging {<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17589/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15726162
--- Diff: project/SparkBuild.scala ---
@@ -62,7 +62,7 @@ object SparkBuild extends PomBuild {
var isAlphaYarn = false
var profiles: mutable.Seq[String] = mutable.Seq.empty
if (Properties.envOrNone("SPARK_GANGLIA_LGPL").isDefined) {
- println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pganglia-lgpl flag.")
+ println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pspark-ganglia-lgpl flag.")
--- End diff --
There is nothing in that JIRA. no monty at all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50512322
QA tests have started for PR 1434. This patch DID NOT merge cleanly! <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17365/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15567310
--- Diff: pom.xml ---
@@ -970,6 +972,14 @@
</modules>
</profile>
+ <!-- Kinesis integration is not included by default due to ASL-licensed code -->
+ <profile>
+ <id>spark-kinesis-asl</id>
--- End diff --
Profile can be named `kinesis-asl`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15670661
--- Diff: dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala ---
@@ -47,7 +47,14 @@ object SimpleApp {
System.exit(-1)
}
if (foundGanglia) {
- println("Ganglia sink was loaded via spark-core")
+ println("Ganglia sink was loaded via spark-ganglia-lgpl")
--- End diff --
This should remain as spark-core. This essentially checks whether dependencies on spark-core should not bring in Ganglia. So the statement is correct.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50253422
also, can someone address the questions i have here regarding the ec2 scripts and other peripheral aspects of this PR: https://issues.apache.org/jira/browse/SPARK-1981?focusedCommentId=14072761
thanks!
-chris
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15563579
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+import java.nio.ByteBuffer
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.
+ *
+ * @param app name
+ * @param Kinesis stream name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * @param persistence strategy for RDDs and DStreams.
+ */
+private[streaming] class KinesisReceiver(
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream,
+ storageLevel: StorageLevel)
+ extends Receiver[Array[Byte]](storageLevel) with Logging { receiver =>
+
+ /**
+ * The lazy val's below will get instantiated in the remote Executor after the closure is shipped to the Spark Worker.
+ * These are all lazy because they're from third-party Amazon libraries and are not Serializable.
+ * If they're not marked lazy, they will cause NotSerializableExceptions when they're shipped to the Spark Worker.
+ */
+
+ /**
+ * workerId is lazy because we want the address of the actual Worker where the code runs - not the Driver's ip address.
+ * This makes a difference when running in a cluster.
+ */
+ lazy val workerId = InetAddress.getLocalHost.getHostAddress() + ":" + UUID.randomUUID()
+
+ /**
+ * This impl uses the DefaultAWSCredentialsProviderChain per the following url:
+ * http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html
+ * and searches for credentials in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
+ * Instance profile credentials delivered through the Amazon EC2 metadata service
+ */
+ lazy val credentialsProvider = new DefaultAWSCredentialsProviderChain()
+
+ /** Create a KCL config instance. */
+ lazy val KinesisClientLibConfiguration = new KinesisClientLibConfiguration(app, stream, credentialsProvider, workerId)
+ .withKinesisEndpoint(endpoint).withInitialPositionInStream(initialPositionInStream).withTaskBackoffTimeMillis(500)
+
+ /**
+ * RecordProcessorFactory creates impls of IRecordProcessor.
+ * IRecordProcessor adapts the KCL to our Spark KinesisReceiver via the IRecordProcessor.processRecords() method.
+ * We're using our custom KinesisRecordProcessor in this case.
+ */
+ lazy val recordProcessorFactory: IRecordProcessorFactory = new IRecordProcessorFactory {
+ override def createProcessor: IRecordProcessor = new KinesisRecordProcessor(receiver, workerId, KinesisUtils.createCheckpointState(checkpointIntervalMillis))
--- End diff --
Line longer than 100 characters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15618561
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCount.scala ---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming
+
+import java.nio.ByteBuffer
+import org.apache.log4j.Level
+import org.apache.log4j.Logger
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkContext._
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer
+import org.apache.spark.streaming.kinesis.KinesisUtils
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.model.PutRecordRequest
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import org.apache.spark.streaming.dstream.DStream
+
+/**
+ * Kinesis Spark Streaming WordCount example.
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the given stream.
+ * It then starts pulling from the tip of the given <stream-name> and <endpoint-url> at the given <batch-interval>.
+ * Because we're pulling from the tip (InitialPositionInStream.LATEST), only new stream data will be picked up after the KinesisReceiver starts.
+ * This could lead to missed records if data is added to the stream while no KinesisReceivers are running.
+ * In production, you'll want to switch to InitialPositionInStream.TRIM_HORIZON which will read up to 24 hours (Kinesis limit) of previous stream data
+ * depending on the checkpoint frequency.
+ *
+ * InitialPositionInStream.TRIM_HORIZON may lead to duplicate processing of records depending on the checkpoint frequency.
+ * Record processing should be idempotent when possible.
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: KinesisWordCount <stream-name> <endpoint-url> <batch-interval>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service (ie. https://kinesis.us-east-1.amazonaws.com)
+ * <batch-interval> is the batch interval in millis (ie. 1000ms)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ bin/run-kinesis-example \
+ * org.apache.spark.examples.streaming.KinesisWordCount mySparkStream https://kinesis.us-east-1.amazonaws.com 100
+ *
+ * There is a companion helper class below called KinesisWordCountProducer which puts dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in that class definition.
+ */
+object KinesisWordCount extends Logging {
+ val WordSeparator = " "
+
+ def main(args: Array[String]) {
+/**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 3) {
+ System.err.println("Usage: KinesisWordCount <stream-name> <endpoint-url> <batch-interval>")
+ System.exit(1)
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements
+ if (!log4jInitialized) {
+ /** We first log something to initialize Spark's default logging, then we override the logging level. */
+ logInfo("Setting log level to [INFO] for streaming example." +
+ " To override add a custom log4j.properties to the classpath.")
+
+ Logger.getRootLogger().setLevel(Level.INFO)
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ val Array(stream, endpoint, batchIntervalMillisStr) = args
+ val batchIntervalMillis = batchIntervalMillisStr.toInt
+
+ /** Create a Kinesis client in order to determine the number of shards for the given stream */
+ val KinesisClient = new AmazonKinesisClient(new DefaultAWSCredentialsProviderChain());
+
+ /** Determine the number of shards from the stream */
+ val numShards = KinesisClient.describeStream(stream).getStreamDescription().getShards().size()
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStreams for each stream shard */
+ val numStreams = numShards
+
+ /** Must add 1 more thread than the number of receivers or the output won't show properly from the driver */
+ val numSparkThreads = numStreams + 1
+
+ /** Set the app name */
+ val app = "KinesisWordCount"
+
+ /** Setup the Spark config. */
+ val sparkConfig = new SparkConf().setAppName(app).setMaster(s"local[$numSparkThreads]")
+
+ /**
+ * Set the batch interval.
+ * Records will be pulled from the Kinesis stream and stored as a single DStream within Spark every batch interval.
+ */
+ val batchInterval = Milliseconds(batchIntervalMillis)
+
+ /**
+ * It's recommended that you perform a Spark checkpoint between 5 and 10 times the batch interval.
+ * While this is the Spark checkpoint interval, we're going to use it for the Kinesis checkpoint interval, as well.
+ */
+ val checkpointInterval = batchInterval * 5
+
+ /** Setup the StreamingContext */
+ val ssc = new StreamingContext(sparkConfig, batchInterval)
+
+ /** Setup the checkpoint directory used by Spark Streaming */
+ ssc.checkpoint("/tmp/checkpoint");
+
+ /** Create the same number of Kinesis Receivers/DStreams as stream shards, then union them all */
+ var allStreams: DStream[Array[Byte]] = KinesisUtils.createStream(ssc, app, stream, endpoint, checkpointInterval.milliseconds,
+ InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2)
+ /** Set the checkpoint interval */
+ allStreams.checkpoint(checkpointInterval)
+ for (i <- 1 until numStreams) {
+ /** Create a new Receiver/DStream for each stream shard */
+ val dStream = KinesisUtils.createStream(ssc, app, stream, endpoint, checkpointInterval.milliseconds,
+ InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2)
+ /** Set the Spark checkpoint interval */
+ dStream.checkpoint(checkpointInterval)
+
+ /** Union with the existing streams */
+ allStreams = allStreams.union(dStream)
+ }
+
+ /** This implementation uses the String-based KinesisRecordSerializer impl */
+ val recordSerializer = new KinesisStringRecordSerializer()
+
+ /**
+ * Sort and print the given dstream.
+ * This is an Output Operation that will materialize the underlying DStream.
+ * Everything up to this point is a lazy Transformation Operation.
+ *
+ * @param description of the dstream for logging purposes
+ * @param dstream to sort and print
+ */
+ def sortAndPrint(description: String, dstream: DStream[(String,Int)]) = {
+ dstream.foreachRDD((batch, endOfWindowTime) => {
--- End diff --
removed the window operation complexity
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15561299
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): ReceiverInputDStream[Array[Byte]] = {
+
+ ssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
--- End diff --
@param initialPositionInStream is missing in the docs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50704814
Chris, thanks for this very thorough PR. One thought I have is that maybe we should put the example in the `examples` package, and make that depend on Kinesis. The Twitter example is in there. Now this would make the examples package depend on ASL, but I don't think anyone links to that directly, or we can even decide not to publish it to Maven. And to differentiate this file from the others, you could add ASL in its file name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15690371
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisStringRecordSerializer.scala ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+
+/**
+ * Implementation of KinesisRecordSerializer to convert Array[Byte] to/from String.
+ */
+class KinesisStringRecordSerializer extends KinesisRecordSerializer[String] with Logging {
--- End diff --
Cool!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15552530
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+import java.nio.ByteBuffer
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.
+ *
+ * @param app name
--- End diff --
What is this app name? Please add to comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15552628
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+import java.nio.ByteBuffer
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.
+ *
+ * @param app name
+ * @param Kinesis stream name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * @param persistence strategy for RDDs and DStreams.
+ */
+private[streaming] class KinesisReceiver(
+ app: String,
--- End diff --
Formatting incorrect. 4 spaces necessary here. Please see the spark style guide
https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15654371
--- Diff: make-distribution.sh ---
@@ -196,6 +196,8 @@ cp -r "$FWDIR/bin" "$DISTDIR"
cp -r "$FWDIR/python" "$DISTDIR"
cp -r "$FWDIR/sbin" "$DISTDIR"
cp -r "$FWDIR/ec2" "$DISTDIR"
+cp -r "$FWDIR/extras/kinesis-asl/bin" "$DISTDIR"
--- End diff --
@pwendell Can you take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15623520
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
--- End diff --
Better to make it more explicit. `Utility / Helper class to create input stream with Amazon Kinesis`
No need to add lower level details like " reusable methods " and stuff
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15618507
--- Diff: project/SparkBuild.scala ---
@@ -60,9 +60,13 @@ object SparkBuild extends PomBuild {
var isAlphaYarn = false
var profiles: mutable.Seq[String] = mutable.Seq.empty
if (Properties.envOrNone("SPARK_GANGLIA_LGPL").isDefined) {
- println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pganglia-lgpl flag.")
+ println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pspark-ganglia-lgpl flag.")
profiles ++= Seq("spark-ganglia-lgpl")
}
+ if (Properties.envOrNone("SPARK_KINESIS_ASL").isDefined) {
--- End diff --
removed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15568976
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): ReceiverInputDStream[Array[Byte]] = {
+
+ ssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return JavaReceiverInputDStream[Array[Byte]]
+ */
+ def createJavaStream(
+ jssc: JavaStreamingContext,
+ app: String,
--- End diff --
incorrect formatting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50250282
Jenkins, this is ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15617751
--- Diff: docs/streaming-programming-guide.md ---
@@ -467,6 +468,62 @@ For more details on these additional sources, see the corresponding [API documen
Furthermore, you can also implement your own custom receiver for your sources. See the
[Custom Receiver Guide](streaming-custom-receivers.html).
+### Kinesis
--- End diff --
moved
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15618503
--- Diff: extras/spark-kinesis-asl/src/test/resources/log4j.properties ---
@@ -0,0 +1,42 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Set everything to be logged to the file streaming/target/unit-tests.log
+log4j.rootCategory=WARN, console
+
--- End diff --
matched
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15549239
--- Diff: bin/run-kinesis-example ---
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+
--- End diff --
@pdeyhim explained me that this script is necessary as the jar for kinesis is not included in the spark jar. That makes sense.
But its best not to put such module-specific scripts in the main spark/bin directory. Its best to put this in `extras/kinesis/bin/` .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15620896
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): ReceiverInputDStream[Array[Byte]] = {
+
+ ssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return JavaReceiverInputDStream[Array[Byte]]
+ */
+ def createJavaStream(
+ jssc: JavaStreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): JavaReceiverInputDStream[Array[Byte]] = {
+
+ jssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create checkpoint state using the existing system clock
+ * @param checkpointIntervalMillis
+ */
+ def createCheckpointState(checkpointIntervalMillis: Long): CheckpointState = {
+ new CheckpointState(checkpointIntervalMillis)
+ }
+
+ /**
+ * Retry the given amount of times with a random backoff time (millis) less than the given maxBackOffMillis
+ *
+ * @param expression expression to evalute
+ * @param numRetriesLeft number of retries left
+ * @param maxBackOffMillis: max millis between retries
+ *
+ * @return Evaluation of the given expression
+ * @throws Unretryable exception, unexpected exception,
+ * or any exception that persists after numRetriesLeft reaches 0
+ */
+ @annotation.tailrec
+ def retry[T](expression: => T, numRetriesLeft: Int, maxBackOffMillis: Int): T = {
--- End diff --
ok, i've moved this method to a private helper class. it's a static method, so from a testability standpoint, it's awkward to have this part of a non-static class such as KinesisReceiver.
otherwise, i would need to instantiate a KinesisReceiver with a bunch of mocks just to test this method.
this seems like a good compromise, but let me know if you feel otherwise.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15141666
--- Diff: project/SparkBuild.scala ---
@@ -63,6 +63,10 @@ object SparkBuild extends PomBuild {
println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pganglia-lgpl flag.")
profiles ++= Seq("spark-ganglia-lgpl")
}
+ if (Properties.envOrNone("SPARK_KINESIS_ASL").isDefined) {
+ println("NOTE: SPARK_KINESIS_ASL is deprecated, please use -Pspark-kinesis-asl flag.")
+ profiles ++= Seq("spark-ganglia-lgpl")
--- End diff --
good catch, stephen. thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50511469
Jenkins, this is ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15555852
--- Diff: extras/spark-kinesis-asl/pom.xml ---
@@ -0,0 +1,98 @@
+<?xml version="1.0" encoding="UTF-8"?>
--- End diff --
makes sense. i'll update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50956263
QA tests have started for PR 1434. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17760/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15568964
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
--- End diff --
Incorrect formatting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15653238
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param appName Kinesis Application Name. Kinesis Apps are mapped to Kinesis Streams
+ * by the Kinesis Client Library. If you change the App name or Stream name,
+ * the KCL will throw errors.
+ * @param stream Kinesis Stream Name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types
+ * of checkpoints.
+ * @param initialPositionInStream in the absence of Kinesis checkpoint info, this is the
+ * worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours
+ * (InitialPositionInStream.TRIM_HORIZON) or the tip of the stream
+ * (InitialPositionInStream.LATEST).
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk
--- End diff --
right, i knew you would ask about this!
scala can't differentiate between the two overloaded methods with default arguments in this situation: https://groups.google.com/forum/#!msg/scala-user/FyQK3-cqfaY/fXLHr8QsW_0J
it's totes weird, i know.
i'll remove storageLevel to simplify. i'd like to keep initialPositionInStream in case there's a need to differentiate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15721167
--- Diff: dev/audit-release/sbt_app_kinesis/build.sbt ---
@@ -0,0 +1,30 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+name := "Kinesis Test"
+
+version := "1.0"
+
+scalaVersion := System.getenv.get("SCALA_VERSION")
+
+libraryDependencies += "org.apache.spark" %% "spark-core" % System.getenv.get("SPARK_VERSION")
+libraryDependencies += "org.apache.spark" %% "spark-streaming" % System.getenv.get("SPARK_VERSION")
--- End diff --
nit: done need spark-core and spark-streaming as they will be caught through dependencies of spark-streaming-kinesis-asl
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15564306
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): ReceiverInputDStream[Array[Byte]] = {
+
+ ssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return JavaReceiverInputDStream[Array[Byte]]
+ */
+ def createJavaStream(
+ jssc: JavaStreamingContext,
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream = InitialPositionInStream.TRIM_HORIZON,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2): JavaReceiverInputDStream[Array[Byte]] = {
+
+ jssc.receiverStream(new KinesisReceiver(app, stream, endpoint, checkpointIntervalMillis, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create checkpoint state using the existing system clock
+ * @param checkpointIntervalMillis
+ */
+ def createCheckpointState(checkpointIntervalMillis: Long): CheckpointState = {
+ new CheckpointState(checkpointIntervalMillis)
--- End diff --
What is the purpose of this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15564529
--- Diff: project/SparkBuild.scala ---
@@ -60,9 +60,13 @@ object SparkBuild extends PomBuild {
var isAlphaYarn = false
var profiles: mutable.Seq[String] = mutable.Seq.empty
if (Properties.envOrNone("SPARK_GANGLIA_LGPL").isDefined) {
- println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pganglia-lgpl flag.")
+ println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pspark-ganglia-lgpl flag.")
profiles ++= Seq("spark-ganglia-lgpl")
}
+ if (Properties.envOrNone("SPARK_KINESIS_ASL").isDefined) {
--- End diff --
This is not necessary! This exists for HIVE because of legacy configuration methods, and is deprecated. Hence it is not necessary for this new one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15564223
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.util.List
+import scala.collection.JavaConversions.asScalaBuffer
+import scala.collection.mutable.ArrayBuffer
+import org.apache.spark.Logging
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.SystemClock
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer
--- End diff --
Please order the imports as recommended by Spark coding guide.
https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50786500
ah, gotcha matei. the examples aren't part of the core, so they can depend on external libs. i'll make the change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15555717
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+import java.nio.ByteBuffer
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.
+ *
+ * @param app name
+ * @param Kinesis stream name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * @param persistence strategy for RDDs and DStreams.
+ */
+private[streaming] class KinesisReceiver(
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream,
+ storageLevel: StorageLevel)
+ extends Receiver[Array[Byte]](storageLevel) with Logging { receiver =>
+
+ /**
+ * The lazy val's below will get instantiated in the remote Executor after the closure is shipped to the Spark Worker.
+ * These are all lazy because they're from third-party Amazon libraries and are not Serializable.
+ * If they're not marked lazy, they will cause NotSerializableExceptions when they're shipped to the Spark Worker.
+ */
+
+ /**
+ * workerId is lazy because we want the address of the actual Worker where the code runs - not the Driver's ip address.
+ * This makes a difference when running in a cluster.
+ */
+ lazy val workerId = InetAddress.getLocalHost.getHostAddress() + ":" + UUID.randomUUID()
--- End diff --
there can be multiple workers per host, so i can't just use the host address. but to answer your question, i guess i don't really need the host address since i'm generating a random UUID.
however, i found it useful when reviewing logs for debugging purposes. i'll keep for now unless you have a strong objection.
good catch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15726898
--- Diff: bin/run-example ---
@@ -29,7 +29,9 @@ if [ -n "$1" ]; then
else
echo "Usage: ./bin/run-example <example-class> [example-args]" 1>&2
echo " - set MASTER=XX to use a specific master" 1>&2
- echo " - can use abbreviated example class name (e.g. SparkPi, mllib.LinearRegression)" 1>&2
+ echo " - can use abbreviated example class name relative to com.apache.spark.examples" 1>&2
+ echo " (e.g. SparkPi, mllib.LinearRegression, streaming.KinesisWordCountASL)" 1>&2
+ echo " - to run the Kinesis Spark Streaming example, make sure you build with -Pkinesis-asl" 1>&2
--- End diff --
Hey I'd prefer not to have this in this short text here. We can add a note in the docs for this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50530784
Hey @cfregly,
My apologies for being so late to review this. Seems like the PR requires merging with the master, probably because of changes to pom.xml. Would you get a chance to merge with the master once again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15723355
--- Diff: extras/kinesis-asl/src/main/resources/log4j.properties ---
@@ -0,0 +1,42 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Set everything to be logged to the file streaming/target/unit-tests.log
--- End diff --
This lines was removed. as this property does not determine that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15726168
--- Diff: project/SparkBuild.scala ---
@@ -62,7 +62,7 @@ object SparkBuild extends PomBuild {
var isAlphaYarn = false
var profiles: mutable.Seq[String] = mutable.Seq.empty
if (Properties.envOrNone("SPARK_GANGLIA_LGPL").isDefined) {
- println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pganglia-lgpl flag.")
+ println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pspark-ganglia-lgpl flag.")
--- End diff --
But I think i get the point, that this is the wrong advice statement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50974810
This was an epic pull request. Nice work, people.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50787956
QA tests have started for PR 1434. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17587/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15564189
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.net.InetAddress
+import java.util.UUID
+import org.apache.spark.Logging
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker
+import java.nio.ByteBuffer
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver.
+ * This implementation relies on the Kinesis Client Library (KCL) Worker as described here:
+ * https://github.com/awslabs/amazon-kinesis-client
+ * This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here:
+ * http://spark.apache.org/docs/latest/streaming-custom-receivers.html
+ * Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.
+ *
+ * @param app name
+ * @param Kinesis stream name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * @param in the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON)
+ * or the tip of the stream using InitialPositionInStream.LATEST.
+ * @param persistence strategy for RDDs and DStreams.
+ */
+private[streaming] class KinesisReceiver(
+ app: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream,
+ storageLevel: StorageLevel)
+ extends Receiver[Array[Byte]](storageLevel) with Logging { receiver =>
+
+ /**
+ * The lazy val's below will get instantiated in the remote Executor after the closure is shipped to the Spark Worker.
+ * These are all lazy because they're from third-party Amazon libraries and are not Serializable.
+ * If they're not marked lazy, they will cause NotSerializableExceptions when they're shipped to the Spark Worker.
+ */
+
+ /**
+ * workerId is lazy because we want the address of the actual Worker where the code runs - not the Driver's ip address.
+ * This makes a difference when running in a cluster.
+ */
+ lazy val workerId = InetAddress.getLocalHost.getHostAddress() + ":" + UUID.randomUUID()
+
+ /**
+ * This impl uses the DefaultAWSCredentialsProviderChain per the following url:
+ * http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html
+ * and searches for credentials in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
+ * Instance profile credentials delivered through the Amazon EC2 metadata service
+ */
+ lazy val credentialsProvider = new DefaultAWSCredentialsProviderChain()
+
+ /** Create a KCL config instance. */
+ lazy val KinesisClientLibConfiguration = new KinesisClientLibConfiguration(app, stream, credentialsProvider, workerId)
+ .withKinesisEndpoint(endpoint).withInitialPositionInStream(initialPositionInStream).withTaskBackoffTimeMillis(500)
--- End diff --
Hey, so this is probably not properly documented, but the receivers can be started and stopped multiple time (forexample, when receiver.restart(<Error>) is called). So creating all these vals once using lazy is not the right way, and then calling run/shutdown multiple times is not a good idea. Instead they should be created from scratch every time onStart() is called.
See the pattern followed by [FlumeReceiver](https://github.com/apache/spark/blob/master/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala#L139).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15567741
--- Diff: extras/spark-kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCount.java ---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.examples.streaming;
+
+import java.util.List;
+import java.util.regex.Pattern;
+
+import org.apache.log4j.Level;
+import org.apache.log4j.Logger;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.api.java.function.Function2;
+import org.apache.spark.api.java.function.PairFunction;
+import org.apache.spark.storage.StorageLevel;
+import org.apache.spark.streaming.Duration;
+import org.apache.spark.streaming.Milliseconds;
+import org.apache.spark.streaming.api.java.JavaDStream;
+import org.apache.spark.streaming.api.java.JavaPairDStream;
+import org.apache.spark.streaming.api.java.JavaStreamingContext;
+import org.apache.spark.streaming.dstream.DStream;
+import org.apache.spark.streaming.kinesis.KinesisRecordSerializer;
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer;
+import org.apache.spark.streaming.kinesis.KinesisUtils;
+
+import scala.Tuple2;
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.kinesis.AmazonKinesisClient;
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
+import com.google.common.base.Optional;
+import com.google.common.collect.Lists;
+
+/**
+ * Java-friendly Kinesis Spark Streaming WordCount example
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the given stream.
+ * It then starts pulling from the tip of the given <stream-name> and <endpoint-url> at the given <batch-interval>.
+ * Because we're pulling from the tip (InitialPositionInStream.LATEST), only new stream data will be picked up after the KinesisReceiver starts.
+ * This could lead to missed records if data is added to the stream while no KinesisReceivers are running.
+ * In production, you'll want to switch to InitialPositionInStream.TRIM_HORIZON which will read up to 24 hours (Kinesis limit) of previous stream data
+ * depending on the checkpoint frequency.
+ * InitialPositionInStream.TRIM_HORIZON may lead to duplicate processing of records depending on the checkpoint frequency.
+ * Record processing should be idempotent when possible.
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: JavaKinesisWordCount <stream-name> <endpoint-url> <batch-interval>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service (ie. https://kinesis.us-east-1.amazonaws.com)
+ * <batch-interval> is the batch interval in milliseconds (ie. 1000ms)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ bin/run-kinesis-example \
+ * org.apache.spark.examples.streaming.JavaKinesisWordCount mySparkStream https://kinesis.us-east-1.amazonaws.com 1000
+ *
+ * There is a companion helper class called KinesisWordCountProducer which puts dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in the class definition.
+ */
+public final class JavaKinesisWordCount {
+ private static final Pattern WORD_SEPARATOR = Pattern.compile(" ");
+ private static final Logger logger = Logger.getLogger(JavaKinesisWordCount.class);
+
+ /**
+ * Make the constructor private to enforce singleton
+ */
+ private JavaKinesisWordCount() {
+ }
+
+ public static void main(String[] args) {
+ /**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 3) {
+ System.err.println("Usage: JavaKinesisWordCount <stream-name> <kinesis-endpoint-url> <batch-interval>");
+ System.exit(1);
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ boolean log4jInitialized = Logger.getRootLogger().getAllAppenders()
+ .hasMoreElements();
+ if (!log4jInitialized) {
+ /** We first log something to initialize Spark's default logging, then we override the logging level. */
+ Logger.getRootLogger()
+ .info("Setting log level to [ERROR] for streaming example."
+ + " To override add a custom log4j.properties to the classpath.");
+ Logger.getRootLogger().setLevel(Level.ERROR);
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ String stream = args[0];
+ String endpoint = args[1];
+ Integer batchIntervalMillis = Integer.valueOf(args[2]);
+
+ /** Create a Kinesis client in order to determine the number of shards for the given stream */
+ AmazonKinesisClient KinesisClient = new AmazonKinesisClient(
+ new DefaultAWSCredentialsProviderChain());
+
+ /** Determine the number of shards from the stream */
+ int numShards = KinesisClient.describeStream(stream)
+ .getStreamDescription().getShards().size();
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStreams for each stream shard */
+ int numStreams = numShards;
+
+ /** Must add 1 more thread than the number of receivers or the output won't show properly from the driver */
+ int numSparkThreads = numStreams + 1;
+
+ /** Set the app name */
+ String app = "KinesisWordCount";
+
+ /** Setup the Spark config. */
+ SparkConf sparkConfig = new SparkConf().setAppName(app).setMaster(
+ "local[" + numSparkThreads + "]");
+
+ /**
+ * Set the batch interval.
+ * Records will be pulled from the Kinesis stream and stored as a single DStream within Spark every batch interval.
+ */
+ Duration batchInterval = Milliseconds.apply(batchIntervalMillis);
+
+ /**
+ * It's recommended that you perform a Spark checkpoint between 5 and 10 times the batch interval.
+ * While this is the Spark checkpoint interval, we're going to use it for the Kinesis checkpoint interval, as well.
+ */
+ Duration checkpointInterval = batchInterval.$times(5);
+
--- End diff --
Please do not use $times(), its not a proper exposed function in Duration class, that should be used in Java. If the batch duration is constant 2 seconds, then the other examples can be set as constants.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15685837
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessorUtils.scala ---
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import scala.util.Random
+
+import org.apache.spark.Logging
+
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+
+
+/**
+ * Helper for the KinesisRecordProcessor.
+ */
+private[kinesis] object KinesisRecordProcessorUtils extends Logging {
--- End diff --
added the companion object
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15504371
--- Diff: bin/run-kinesis-example ---
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+
--- End diff --
Why is there a need for a complete script to run the kinesis example? Shouldnt the standard bin/run-example be sufficient?
The thing is that we want to have one streamlined way for running all the examples. The more special cases we make, the more we have to explicitly maintain in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15727688
--- Diff: examples/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming
+
+import java.nio.ByteBuffer
+
+import scala.util.Random
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
+import org.apache.spark.streaming.kinesis.KinesisUtils
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.model.PutRecordRequest
+
+/**
+ * Kinesis Spark Streaming WordCount example.
+ *
+ * See http://spark.apache.org/docs/latest/streaming-kinesis.html for more details on
+ * the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receiver) per shard
+ * for the given stream.
+ * It then starts pulling from the last checkpointed sequence number of the given
+ * <stream-name> and <endpoint-url>.
+ *
+ * Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials
+ * in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: KinesisWordCountASL <stream-name> <endpoint-url>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service
+ * (ie. https://kinesis.us-east-1.amazonaws.com)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ $SPARK_HOME/bin/run-example \
+ * org.apache.spark.examples.streaming.KinesisWordCountASL mySparkStream \
+ * https://kinesis.us-east-1.amazonaws.com
+ *
+ * There is a companion helper class below called KinesisWordCountProducerASL which puts
+ * dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducerASL are provided in that class definition.
+ */
+object KinesisWordCountASL extends Logging {
+ def main(args: Array[String]) {
+/**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 2) {
+ System.err.println(
+ """
+ |Usage: KinesisWordCount <stream-name> <endpoint-url>
+ | <stream-name> is the name of the Kinesis stream
+ | <endpoint-url> is the endpoint of the Kinesis service
+ | (e.g. https://kinesis.us-east-1.amazonaws.com)
+ """.stripMargin)
+ System.exit(1)
+ }
+
+ StreamingExamples.setStreamingLogLevels()
+
+ /** Populate the appropriate variables from the given args */
+ val Array(streamName, endpointUrl) = args
+
+ /** Determine the number of shards from the stream */
+ val kinesisClient = new AmazonKinesisClient(new DefaultAWSCredentialsProviderChain())
+ kinesisClient.setEndpoint(endpointUrl)
+ val numShards = kinesisClient.describeStream(streamName).getStreamDescription().getShards()
+ .size()
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard. */
+ val numStreams = numShards
+
+ /**
+ * numSparkThreads should be 1 more thread than the number of receivers.
+ * This leaves one thread available for actually processing the data.
+ */
+ val numSparkThreads = numStreams + 1
+
+ /** Setup the and SparkConfig and StreamingContext */
+ /** Spark Streaming batch interval */
+ val batchInterval = Milliseconds(2000)
+ val sparkConfig = new SparkConf().setAppName("KinesisWordCount")
+ .setMaster(s"local[$numSparkThreads]")
+ val ssc = new StreamingContext(sparkConfig, batchInterval)
+ /** Setup the checkpoint directory used by Spark Streaming */
+ ssc.checkpoint("/tmp/checkpoint");
--- End diff --
removed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15558055
--- Diff: docs/streaming-programming-guide.md ---
@@ -467,6 +468,62 @@ For more details on these additional sources, see the corresponding [API documen
Furthermore, you can also implement your own custom receiver for your sources. See the
[Custom Receiver Guide](streaming-custom-receivers.html).
+### Kinesis
--- End diff --
This is not the best place to have add documentation about Kinesis. I am planning to do a bit of a overhaul and add sections on Flume, Kafka, etc., though I still havent thought it through. Till then, can you add this documentation in a separate streaming-kinesis.md file, and I will incorporate it in my updates later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50819992
@pwendell Can you take a look at the pom, and other build stuff?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15568250
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCount.scala ---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming
+
+import java.nio.ByteBuffer
+import org.apache.log4j.Level
+import org.apache.log4j.Logger
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkContext._
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer
+import org.apache.spark.streaming.kinesis.KinesisUtils
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.model.PutRecordRequest
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import org.apache.spark.streaming.dstream.DStream
+
+/**
+ * Kinesis Spark Streaming WordCount example.
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the given stream.
+ * It then starts pulling from the tip of the given <stream-name> and <endpoint-url> at the given <batch-interval>.
+ * Because we're pulling from the tip (InitialPositionInStream.LATEST), only new stream data will be picked up after the KinesisReceiver starts.
+ * This could lead to missed records if data is added to the stream while no KinesisReceivers are running.
+ * In production, you'll want to switch to InitialPositionInStream.TRIM_HORIZON which will read up to 24 hours (Kinesis limit) of previous stream data
+ * depending on the checkpoint frequency.
+ *
+ * InitialPositionInStream.TRIM_HORIZON may lead to duplicate processing of records depending on the checkpoint frequency.
+ * Record processing should be idempotent when possible.
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: KinesisWordCount <stream-name> <endpoint-url> <batch-interval>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service (ie. https://kinesis.us-east-1.amazonaws.com)
+ * <batch-interval> is the batch interval in millis (ie. 1000ms)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ bin/run-kinesis-example \
+ * org.apache.spark.examples.streaming.KinesisWordCount mySparkStream https://kinesis.us-east-1.amazonaws.com 100
+ *
+ * There is a companion helper class below called KinesisWordCountProducer which puts dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in that class definition.
+ */
+object KinesisWordCount extends Logging {
+ val WordSeparator = " "
+
+ def main(args: Array[String]) {
+/**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 3) {
+ System.err.println("Usage: KinesisWordCount <stream-name> <endpoint-url> <batch-interval>")
+ System.exit(1)
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements
+ if (!log4jInitialized) {
+ /** We first log something to initialize Spark's default logging, then we override the logging level. */
+ logInfo("Setting log level to [INFO] for streaming example." +
+ " To override add a custom log4j.properties to the classpath.")
+
+ Logger.getRootLogger().setLevel(Level.INFO)
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ val Array(stream, endpoint, batchIntervalMillisStr) = args
+ val batchIntervalMillis = batchIntervalMillisStr.toInt
--- End diff --
Just like the comment in JavaKinesisWordCount, please make the batch duration constant, 2 seconds, and other intervals constant as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15651823
--- Diff: project/SparkBuild.scala ---
@@ -62,7 +62,7 @@ object SparkBuild extends PomBuild {
var isAlphaYarn = false
var profiles: mutable.Seq[String] = mutable.Seq.empty
if (Properties.envOrNone("SPARK_GANGLIA_LGPL").isDefined) {
- println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pganglia-lgpl flag.")
+ println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pspark-ganglia-lgpl flag.")
--- End diff --
that -P wasn't working. new jira for the full monty: https://issues.apache.org/jira/browse/SPARK-2770
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15685855
--- Diff: examples/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCount.scala ---
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming
+
+import java.nio.ByteBuffer
+
+import scala.util.Random
+
+import org.apache.log4j.Level
+import org.apache.log4j.Logger
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkContext.rddToOrderedRDDFunctions
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer
+import org.apache.spark.streaming.kinesis.KinesisUtils
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.model.PutRecordRequest
+
+/**
+ * Kinesis Spark Streaming WordCount example.
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on
+ * the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the
+ * given stream.
+ * It then starts pulling from the last checkpointed sequence number of the given
+ * <stream-name> and <endpoint-url>.
+ *
+ * Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials
+ * in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: KinesisWordCount <stream-name> <endpoint-url>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service
+ * (ie. https://kinesis.us-east-1.amazonaws.com)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ $SPARK_HOME/bin/run-example \
+ * org.apache.spark.examples.streaming.KinesisWordCount mySparkStream \
+ * https://kinesis.us-east-1.amazonaws.com
+ *
+ * There is a companion helper class below called KinesisWordCountProducer which puts
+ * dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in that class definition.
+ */
+object KinesisWordCount extends Logging {
+ val WordSeparator = " "
+
+ def main(args: Array[String]) {
+/**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 2) {
+ System.err.println("Usage: KinesisWordCount <stream-name> <endpoint-url>")
+ System.exit(1)
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency
+ * on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements()
+ if (!log4jInitialized) {
+ /**
+ * We first log something to initialize Spark's default logging,
+ * then we override the logging level.
+ * */
+ logInfo("Setting log level to [INFO] for streaming example." +
+ " To override add a custom log4j.properties to the classpath.")
+
+ Logger.getRootLogger().setLevel(Level.INFO)
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ val Array(stream, endpoint) = args
+ val batchIntervalMillis = 2000
+
+ /** Create a Kinesis client in order to determine the number of shards for the given stream */
+ val kinesisClient = new AmazonKinesisClient(new DefaultAWSCredentialsProviderChain())
+ kinesisClient.setEndpoint(endpoint)
+
+ /** Determine the number of shards from the stream */
+ val numShards = kinesisClient.describeStream(stream).getStreamDescription().getShards().size()
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard.*/
+ val numStreams = numShards
+
+ /**
+ * Must add 1 more thread than the number of receivers or the output won't show properly
+ * from the driver
+ */
+ val numSparkThreads = numStreams + 1
+
+ /** Set the app name */
+ val appName = "KinesisWordCount"
+
+ /** Setup the Spark config. */
+ val sparkConfig = new SparkConf().setAppName(appName).setMaster(s"local[$numSparkThreads]")
+
+ /**
+ * Set the batch interval.
+ * Records will be pulled from the Kinesis stream and stored as a single DStream within Spark
+ * every batch interval.
+ */
+ val batchInterval = Milliseconds(batchIntervalMillis)
+
+ /**
+ * It's recommended that you perform a Spark checkpoint between 5 and 10 times the batch
+ * interval.
+ * While this is the Spark checkpoint interval, we're going to use it for the Kinesis
+ * checkpoint interval, as well.
+ * For example purposes, we'll just use the batchInterval.
+ */
+ val checkpointInterval = batchInterval
+
+ /** Setup the StreamingContext */
+ val ssc = new StreamingContext(sparkConfig, batchInterval)
+
+ /** Setup the checkpoint directory used by Spark Streaming */
+ ssc.checkpoint("/tmp/checkpoint");
+
+ /**
+ * Create the same number of Kinesis Receivers/DStreams as stream shards, then union
+ * them all.
+ */
+ var allStreams: DStream[Array[Byte]] = KinesisUtils.createStream(ssc, appName, stream,
+ endpoint, checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the checkpoint interval */
+ allStreams.checkpoint(checkpointInterval)
+ for (i <- 1 until numStreams) {
+ /** Create a new Receiver/DStream for each stream shard */
+ val dStream = KinesisUtils.createStream(ssc, appName, stream, endpoint,
+ checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the Spark checkpoint interval */
+ dStream.checkpoint(checkpointInterval)
+
+ /** Union with the existing streams */
+ allStreams = allStreams.union(dStream)
+ }
+
+ /** This implementation uses the String-based KinesisRecordSerializer impl */
+ val recordSerializer = new KinesisStringRecordSerializer()
+
+ /**
+ * Sort and print the given dstream.
+ * This is an Output Operation that will materialize the underlying DStream.
+ * Everything up to this point is a lazy Transformation Operation.
+ *
+ * @param description of the dstream for logging purposes
+ * @param dstream to sort and print
+ */
+ def sortAndPrint(description: String, dstream: DStream[(String,Int)]) = {
+ dstream.foreachRDD((batch, endOfWindowTime) => {
+ val sortedBatch = batch.sortByKey(true)
+ logInfo(s"$description @ $endOfWindowTime")
+ sortedBatch.collect().foreach(
+ wordCount => logInfo(s"$wordCount"))
+ }
+ )
+ }
+
+ /**
+ * Split each line of the union'd DStreams into multiple words using flatMap
+ * to produce the collection.
--- End diff --
simplified similar to KafkaWordCount
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by srosenthal <gi...@git.apache.org>.
Github user srosenthal commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15137055
--- Diff: project/SparkBuild.scala ---
@@ -63,6 +63,10 @@ object SparkBuild extends PomBuild {
println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pganglia-lgpl flag.")
profiles ++= Seq("spark-ganglia-lgpl")
}
+ if (Properties.envOrNone("SPARK_KINESIS_ASL").isDefined) {
+ println("NOTE: SPARK_KINESIS_ASL is deprecated, please use -Pspark-kinesis-asl flag.")
+ profiles ++= Seq("spark-ganglia-lgpl")
--- End diff --
Looks like a copy-paste error? Did you mean to use "spark-kinesis-asl" here instead of "spark-ganglia-lgpl"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15727587
--- Diff: extras/kinesis-asl/pom.xml ---
@@ -0,0 +1,99 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+~ Licensed to the Apache Software Foundation (ASF) under one or more
+~ contributor license agreements. See the NOTICE file distributed with
+~ this work for additional information regarding copyright ownership.
+~ The ASF licenses this file to You under the Apache License, Version 2.0
+~ (the "License"); you may not use this file except in compliance with
+~ the License. You may obtain a copy of the License at
+~
+~ http://www.apache.org/licenses/LICENSE-2.0
+~
+~ Unless required by applicable law or agreed to in writing, software
+~ distributed under the License is distributed on an "AS IS" BASIS,
+~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+~ See the License for the specific language governing permissions and
+~ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-parent</artifactId>
+ <version>1.1.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+
+ <!--
+ Kinesis integration is not included by default due to ASL-licensed code.
+ Note: This project - if activated - is packaged with the main Spark assembly.
--- End diff --
oh yikes. yeah, that's just plain wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15567656
--- Diff: extras/spark-kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCount.java ---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.examples.streaming;
+
+import java.util.List;
+import java.util.regex.Pattern;
+
+import org.apache.log4j.Level;
+import org.apache.log4j.Logger;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.api.java.function.Function2;
+import org.apache.spark.api.java.function.PairFunction;
+import org.apache.spark.storage.StorageLevel;
+import org.apache.spark.streaming.Duration;
+import org.apache.spark.streaming.Milliseconds;
+import org.apache.spark.streaming.api.java.JavaDStream;
+import org.apache.spark.streaming.api.java.JavaPairDStream;
+import org.apache.spark.streaming.api.java.JavaStreamingContext;
+import org.apache.spark.streaming.dstream.DStream;
+import org.apache.spark.streaming.kinesis.KinesisRecordSerializer;
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer;
+import org.apache.spark.streaming.kinesis.KinesisUtils;
+
+import scala.Tuple2;
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.kinesis.AmazonKinesisClient;
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
+import com.google.common.base.Optional;
+import com.google.common.collect.Lists;
+
+/**
+ * Java-friendly Kinesis Spark Streaming WordCount example
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the given stream.
+ * It then starts pulling from the tip of the given <stream-name> and <endpoint-url> at the given <batch-interval>.
+ * Because we're pulling from the tip (InitialPositionInStream.LATEST), only new stream data will be picked up after the KinesisReceiver starts.
+ * This could lead to missed records if data is added to the stream while no KinesisReceivers are running.
+ * In production, you'll want to switch to InitialPositionInStream.TRIM_HORIZON which will read up to 24 hours (Kinesis limit) of previous stream data
+ * depending on the checkpoint frequency.
+ * InitialPositionInStream.TRIM_HORIZON may lead to duplicate processing of records depending on the checkpoint frequency.
+ * Record processing should be idempotent when possible.
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: JavaKinesisWordCount <stream-name> <endpoint-url> <batch-interval>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service (ie. https://kinesis.us-east-1.amazonaws.com)
+ * <batch-interval> is the batch interval in milliseconds (ie. 1000ms)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ bin/run-kinesis-example \
+ * org.apache.spark.examples.streaming.JavaKinesisWordCount mySparkStream https://kinesis.us-east-1.amazonaws.com 1000
+ *
+ * There is a companion helper class called KinesisWordCountProducer which puts dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in the class definition.
+ */
+public final class JavaKinesisWordCount {
+ private static final Pattern WORD_SEPARATOR = Pattern.compile(" ");
+ private static final Logger logger = Logger.getLogger(JavaKinesisWordCount.class);
+
+ /**
+ * Make the constructor private to enforce singleton
+ */
+ private JavaKinesisWordCount() {
+ }
+
+ public static void main(String[] args) {
+ /**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 3) {
+ System.err.println("Usage: JavaKinesisWordCount <stream-name> <kinesis-endpoint-url> <batch-interval>");
+ System.exit(1);
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ boolean log4jInitialized = Logger.getRootLogger().getAllAppenders()
+ .hasMoreElements();
+ if (!log4jInitialized) {
+ /** We first log something to initialize Spark's default logging, then we override the logging level. */
+ Logger.getRootLogger()
+ .info("Setting log level to [ERROR] for streaming example."
+ + " To override add a custom log4j.properties to the classpath.");
+ Logger.getRootLogger().setLevel(Level.ERROR);
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ String stream = args[0];
+ String endpoint = args[1];
+ Integer batchIntervalMillis = Integer.valueOf(args[2]);
--- End diff --
Can you make the batch interval a fixed duration? This stays consistent with most other streaming examples. 2 seconds should be good enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15670918
--- Diff: assembly/pom.xml ---
@@ -186,6 +186,16 @@
</dependencies>
</profile>
<profile>
+ <id>kinesis-asl</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>kinesis-asl_${scala.binary.version}</artifactId>
--- End diff --
The artifact id should `spark-streaming-kinesis-asl` because the published maven dependencies should have spark (to make it clear that its related to Spark).
The profile id can be `kinesis-asl` as it is used only at Spark compile time when the person is already dealing with spark.
Sorry if this wasnt clear.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50938943
QA results for PR 1434:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>public final class JavaKinesisWordCountASL {<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17701/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15727599
--- Diff: extras/kinesis-asl/pom.xml ---
@@ -0,0 +1,99 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+~ Licensed to the Apache Software Foundation (ASF) under one or more
+~ contributor license agreements. See the NOTICE file distributed with
+~ this work for additional information regarding copyright ownership.
+~ The ASF licenses this file to You under the Apache License, Version 2.0
+~ (the "License"); you may not use this file except in compliance with
+~ the License. You may obtain a copy of the License at
+~
+~ http://www.apache.org/licenses/LICENSE-2.0
+~
+~ Unless required by applicable law or agreed to in writing, software
+~ distributed under the License is distributed on an "AS IS" BASIS,
+~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+~ See the License for the specific language governing permissions and
+~ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-parent</artifactId>
+ <version>1.1.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+
+ <!--
+ Kinesis integration is not included by default due to ASL-licensed code.
+ Note: This project - if activated - is packaged with the main Spark assembly.
--- End diff --
Correction, this is not a nit. Its already confusing ... please change this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15561324
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import org.apache.spark.Logging
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import scala.util.Random
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.SystemClock
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ */
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param app name
+ * @param stream name
+ * @param endpoint
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk of processing records more than once.
--- End diff --
`@param initialPositionInStream` is incorrect in the docs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50253400
@mateiz -
this is a completely brand-new, from-scratch implementation.
parviz's old code was actually a Scala port of the Java-based Kinesis sample app found here: https://github.com/aws/aws-sdk-java/blob/master/src/samples/AmazonKinesisApplication/SampleRecordProcessor.java
this was fine for a quick/dirty sample of kinesis functionality, but my goal was to make this code more reusable, testable, readable, configurable, production-ready, and well-documented. the old code did not support the new Streaming 1.0 API and extras/ build structure (due to ASL-license). i also updated the AWS Java SDK and Kinesis Client Libraries to their latest versions.
here is parviz's PR for reference/comparison: https://github.com/apache/spark/pull/223.
i've addressed all of the comments provided in the old PR to speed up the acceptance of this new PR.
thanks and i look forward to getting this merged!
-chris
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15504519
--- Diff: extras/spark-kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCount.java ---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.examples.streaming;
+
+import java.util.List;
+import java.util.regex.Pattern;
+
+import org.apache.log4j.Level;
+import org.apache.log4j.Logger;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.api.java.function.Function2;
+import org.apache.spark.api.java.function.PairFunction;
+import org.apache.spark.storage.StorageLevel;
+import org.apache.spark.streaming.Duration;
+import org.apache.spark.streaming.Milliseconds;
+import org.apache.spark.streaming.api.java.JavaDStream;
+import org.apache.spark.streaming.api.java.JavaPairDStream;
+import org.apache.spark.streaming.api.java.JavaStreamingContext;
+import org.apache.spark.streaming.dstream.DStream;
+import org.apache.spark.streaming.kinesis.KinesisRecordSerializer;
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer;
+import org.apache.spark.streaming.kinesis.KinesisUtils;
+
+import scala.Tuple2;
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.kinesis.AmazonKinesisClient;
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
+import com.google.common.base.Optional;
+import com.google.common.collect.Lists;
+
+/**
+ * Java-friendly Kinesis Spark Streaming WordCount example
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the given stream.
+ * It then starts pulling from the tip of the given <stream-name> and <endpoint-url> at the given <batch-interval>.
+ * Because we're pulling from the tip (InitialPositionInStream.LATEST), only new stream data will be picked up after the KinesisReceiver starts.
+ * This could lead to missed records if data is added to the stream while no KinesisReceivers are running.
+ * In production, you'll want to switch to InitialPositionInStream.TRIM_HORIZON which will read up to 24 hours (Kinesis limit) of previous stream data
+ * depending on the checkpoint frequency.
+ * InitialPositionInStream.TRIM_HORIZON may lead to duplicate processing of records depending on the checkpoint frequency.
+ * Record processing should be idempotent when possible.
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: JavaKinesisWordCount <stream-name> <endpoint-url> <batch-interval>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service (ie. https://kinesis.us-east-1.amazonaws.com)
+ * <batch-interval> is the batch interval in milliseconds (ie. 1000ms)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ bin/run-kinesis-example \
+ * org.apache.spark.examples.streaming.JavaKinesisWordCount mySparkStream https://kinesis.us-east-1.amazonaws.com 1000
+ *
+ * There is a companion helper class called KinesisWordCountProducer which puts dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in the class definition.
+ */
+public final class JavaKinesisWordCount {
+ private static final Pattern WORD_SEPARATOR = Pattern.compile(" ");
+ private static final Logger logger = Logger.getLogger(JavaKinesisWordCount.class);
+
+ /**
+ * Make the constructor private to enforce singleton
+ */
+ private JavaKinesisWordCount() {
+ }
+
+ public static void main(String[] args) {
+ /**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 3) {
+ System.err.println("Usage: JavaKinesisWordCount <stream-name> <kinesis-endpoint-url> <batch-interval>");
+ System.exit(1);
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ boolean log4jInitialized = Logger.getRootLogger().getAllAppenders()
+ .hasMoreElements();
+ if (!log4jInitialized) {
+ /** We first log something to initialize Spark's default logging, then we override the logging level. */
+ Logger.getRootLogger()
+ .info("Setting log level to [ERROR] for streaming example."
+ + " To override add a custom log4j.properties to the classpath.");
+ Logger.getRootLogger().setLevel(Level.ERROR);
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ String stream = args[0];
+ String endpoint = args[1];
+ Integer batchIntervalMillis = Integer.valueOf(args[2]);
+
+ /** Create a Kinesis client in order to determine the number of shards for the given stream */
+ AmazonKinesisClient KinesisClient = new AmazonKinesisClient(
+ new DefaultAWSCredentialsProviderChain());
+
+ /** Determine the number of shards from the stream */
+ int numShards = KinesisClient.describeStream(stream)
+ .getStreamDescription().getShards().size();
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStreams for each stream shard */
+ int numStreams = numShards;
+
+ /** Must add 1 more thread than the number of receivers or the output won't show properly from the driver */
+ int numSparkThreads = numStreams + 1;
+
+ /** Set the app name */
+ String app = "KinesisWordCount";
+
+ /** Setup the Spark config. */
+ SparkConf sparkConfig = new SparkConf().setAppName(app).setMaster(
+ "local[" + numSparkThreads + "]");
+
+ /**
+ * Set the batch interval.
+ * Records will be pulled from the Kinesis stream and stored as a single DStream within Spark every batch interval.
+ */
+ Duration batchInterval = Milliseconds.apply(batchIntervalMillis);
+
+ /**
+ * It's recommended that you perform a Spark checkpoint between 5 and 10 times the batch interval.
+ * While this is the Spark checkpoint interval, we're going to use it for the Kinesis checkpoint interval, as well.
+ */
+ Duration checkpointInterval = batchInterval.$times(5);
+
+ /** Setup the StreamingContext */
+ JavaStreamingContext jssc = new JavaStreamingContext(sparkConfig, batchInterval);
+
+ /** Setup the checkpoint directory used by Spark Streaming */
+ jssc.checkpoint("/tmp/checkpoint");
+
+ /** Create the same number of Kinesis Receivers/DStreams as stream shards, then union them all */
+ JavaDStream<byte[]> allStreams = KinesisUtils
+ .createJavaStream(jssc, app, stream, endpoint, checkpointInterval.milliseconds(),
+ InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2());
+ /** Set the checkpoint interval */
+ allStreams.checkpoint(checkpointInterval);
+ for (int i = 1; i < numStreams; i++) {
+ /** Create a new Receiver/DStream for each stream shard */
+ JavaDStream<byte[]> dStream = KinesisUtils
+ .createJavaStream(jssc, app, stream, endpoint, checkpointInterval.milliseconds(),
+ InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2());
+ /** Set the Spark checkpoint interval */
+ dStream.checkpoint(checkpointInterval);
+
+ /** Union with the existing streams */
+ allStreams = allStreams.union(dStream);
+ }
+
+ /** This implementation uses the String-based KinesisRecordSerializer impl */
+ final KinesisRecordSerializer<String> recordSerializer = new KinesisStringRecordSerializer();
+
+ /**
+ * Split each line of the union'd DStreams into multiple words using flatMap to produce the collection.
+ * Convert lines of byte[] to multiple Strings by first converting to String, then splitting on WORD_SEPARATOR
+ * We're caching the result here so that we can use it later without having to re-materialize the underlying RDDs.
+ */
+ JavaDStream<String> words = allStreams
+ .flatMap(new FlatMapFunction<byte[], String>() {
+ /**
--- End diff --
Better line break is ...
... allStreams.flatMap( // new line here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15724783
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Duration
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Helper class to create Amazon Kinesis Input Stream
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param ssc StreamingContext object
+ * @param streamName Kinesis stream name
+ * @param endpointUrl Url of Kinesis service (e.g., https://kinesis.us-east-1.amazonaws.com)
+ * @param checkpointInterval Checkpoint interval for Kinesis checkpointing.
+ * See the Kinesis Spark Streaming documentation for more
+ * details on the different types of checkpoints.
+ * @param initialPositionInStream In the absence of Kinesis checkpoint info, this is the
+ * worker's initial starting position in the stream.
+ * The values are either the beginning of the stream
+ * per Kinesis' limit of 24 hours
+ * (InitialPositionInStream.TRIM_HORIZON) or
+ * the tip of the stream (InitialPositionInStream.LATEST).
+ * @param storageLevel Storage level to use for storing the received objects
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ streamName: String,
+ endpointUrl: String,
+ checkpointInterval: Duration,
+ initialPositionInStream: InitialPositionInStream,
+ storageLevel: StorageLevel): ReceiverInputDStream[Array[Byte]] = {
+ ssc.receiverStream(new KinesisReceiver(ssc.sc.appName, streamName, endpointUrl,
+ checkpointInterval, initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param jssc Java StreamingContext object
+ * @param ssc StreamingContext object
+ * @param streamName Kinesis stream name
+ * @param endpointUrl Url of Kinesis service (e.g., https://kinesis.us-east-1.amazonaws.com)
+ * @param checkpointInterval Checkpoint interval for Kinesis checkpointing.
+ * See the Kinesis Spark Streaming documentation for more
+ * details on the different types of checkpoints.
+ * @param initialPositionInStream In the absence of Kinesis checkpoint info, this is the
+ * worker's initial starting position in the stream.
+ * The values are either the beginning of the stream
+ * per Kinesis' limit of 24 hours
+ * (InitialPositionInStream.TRIM_HORIZON) or
+ * the tip of the stream (InitialPositionInStream.LATEST).
+ * @param storageLevel Storage level to use for storing the received objects
+ *
+ * @return JavaReceiverInputDStream[Array[Byte]]
+ *
+ * @return JavaReceiverInputDStream[Array[Byte]]
--- End diff --
Duplicate!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15727580
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Duration
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Helper class to create Amazon Kinesis Input Stream
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
--- End diff --
removed and cleaned up imports
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15623331
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
--- End diff --
Should `@param ssc StreamingContext object`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15668060
--- Diff: examples/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCount.scala ---
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming
+
+import java.nio.ByteBuffer
+
+import scala.util.Random
+
+import org.apache.log4j.Level
+import org.apache.log4j.Logger
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkContext.rddToOrderedRDDFunctions
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.streaming.kinesis.KinesisStringRecordSerializer
+import org.apache.spark.streaming.kinesis.KinesisUtils
+
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.model.PutRecordRequest
+
+/**
+ * Kinesis Spark Streaming WordCount example.
+ *
+ * See http://spark.apache.org/docs/latest/streaming-programming-guide.html for more details on
+ * the Kinesis Spark Streaming integration.
+ *
+ * This example spins up 1 Kinesis Worker (Spark Streaming Receivers) per shard of the
+ * given stream.
+ * It then starts pulling from the last checkpointed sequence number of the given
+ * <stream-name> and <endpoint-url>.
+ *
+ * Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region
+ *
+ * This code uses the DefaultAWSCredentialsProviderChain and searches for credentials
+ * in the following order of precedence:
+ * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
+ * Java System Properties - aws.accessKeyId and aws.secretKey
+ * Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs
+ * Instance profile credentials - delivered through the Amazon EC2 metadata service
+ *
+ * Usage: KinesisWordCount <stream-name> <endpoint-url>
+ * <stream-name> is the name of the Kinesis stream (ie. mySparkStream)
+ * <endpoint-url> is the endpoint of the Kinesis service
+ * (ie. https://kinesis.us-east-1.amazonaws.com)
+ *
+ * Example:
+ * $ export AWS_ACCESS_KEY_ID=<your-access-key>
+ * $ export AWS_SECRET_KEY=<your-secret-key>
+ * $ $SPARK_HOME/bin/run-example \
+ * org.apache.spark.examples.streaming.KinesisWordCount mySparkStream \
+ * https://kinesis.us-east-1.amazonaws.com
+ *
+ * There is a companion helper class below called KinesisWordCountProducer which puts
+ * dummy data onto the Kinesis stream.
+ * Usage instructions for KinesisWordCountProducer are provided in that class definition.
+ */
+object KinesisWordCount extends Logging {
+ val WordSeparator = " "
+
+ def main(args: Array[String]) {
+/**
+ * Check that all required args were passed in.
+ */
+ if (args.length < 2) {
+ System.err.println("Usage: KinesisWordCount <stream-name> <endpoint-url>")
+ System.exit(1)
+ }
+
+ /**
+ * (This was lifted from the StreamingExamples.scala in order to avoid the dependency
+ * on the spark-examples artifact.)
+ * Set reasonable logging levels for streaming if the user has not configured log4j.
+ */
+ val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements()
+ if (!log4jInitialized) {
+ /**
+ * We first log something to initialize Spark's default logging,
+ * then we override the logging level.
+ * */
+ logInfo("Setting log level to [INFO] for streaming example." +
+ " To override add a custom log4j.properties to the classpath.")
+
+ Logger.getRootLogger().setLevel(Level.INFO)
+ Logger.getLogger("org.apache.spark.examples.streaming").setLevel(Level.DEBUG);
+ }
+
+ /** Populate the appropriate variables from the given args */
+ val Array(stream, endpoint) = args
+ val batchIntervalMillis = 2000
+
+ /** Create a Kinesis client in order to determine the number of shards for the given stream */
+ val kinesisClient = new AmazonKinesisClient(new DefaultAWSCredentialsProviderChain())
+ kinesisClient.setEndpoint(endpoint)
+
+ /** Determine the number of shards from the stream */
+ val numShards = kinesisClient.describeStream(stream).getStreamDescription().getShards().size()
+
+ /** In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard.*/
+ val numStreams = numShards
+
+ /**
+ * Must add 1 more thread than the number of receivers or the output won't show properly
+ * from the driver
+ */
+ val numSparkThreads = numStreams + 1
+
+ /** Set the app name */
+ val appName = "KinesisWordCount"
+
+ /** Setup the Spark config. */
+ val sparkConfig = new SparkConf().setAppName(appName).setMaster(s"local[$numSparkThreads]")
+
+ /**
+ * Set the batch interval.
+ * Records will be pulled from the Kinesis stream and stored as a single DStream within Spark
+ * every batch interval.
+ */
+ val batchInterval = Milliseconds(batchIntervalMillis)
+
+ /**
+ * It's recommended that you perform a Spark checkpoint between 5 and 10 times the batch
+ * interval.
+ * While this is the Spark checkpoint interval, we're going to use it for the Kinesis
+ * checkpoint interval, as well.
+ * For example purposes, we'll just use the batchInterval.
+ */
+ val checkpointInterval = batchInterval
+
+ /** Setup the StreamingContext */
+ val ssc = new StreamingContext(sparkConfig, batchInterval)
+
+ /** Setup the checkpoint directory used by Spark Streaming */
+ ssc.checkpoint("/tmp/checkpoint");
+
+ /**
+ * Create the same number of Kinesis Receivers/DStreams as stream shards, then union
+ * them all.
+ */
+ var allStreams: DStream[Array[Byte]] = KinesisUtils.createStream(ssc, appName, stream,
+ endpoint, checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the checkpoint interval */
+ allStreams.checkpoint(checkpointInterval)
+ for (i <- 1 until numStreams) {
+ /** Create a new Receiver/DStream for each stream shard */
+ val dStream = KinesisUtils.createStream(ssc, appName, stream, endpoint,
+ checkpointInterval.milliseconds, InitialPositionInStream.LATEST)
+ /** Set the Spark checkpoint interval */
+ dStream.checkpoint(checkpointInterval)
+
+ /** Union with the existing streams */
+ allStreams = allStreams.union(dStream)
+ }
+
+ /** This implementation uses the String-based KinesisRecordSerializer impl */
+ val recordSerializer = new KinesisStringRecordSerializer()
+
+ /**
+ * Sort and print the given dstream.
+ * This is an Output Operation that will materialize the underlying DStream.
+ * Everything up to this point is a lazy Transformation Operation.
+ *
+ * @param description of the dstream for logging purposes
+ * @param dstream to sort and print
+ */
+ def sortAndPrint(description: String, dstream: DStream[(String,Int)]) = {
+ dstream.foreachRDD((batch, endOfWindowTime) => {
+ val sortedBatch = batch.sortByKey(true)
+ logInfo(s"$description @ $endOfWindowTime")
+ sortedBatch.collect().foreach(
+ wordCount => logInfo(s"$wordCount"))
+ }
+ )
+ }
+
+ /**
+ * Split each line of the union'd DStreams into multiple words using flatMap
+ * to produce the collection.
--- End diff --
I know you simplified this, but I would prefer this to be even more simpler. Its better to have small examples which shows one very simple thing. And then people can combine this examples / components. Take a look at
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/KafkaWordCount.scala
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50957227
QA results for PR 1434:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>public final class JavaKinesisWordCountASL {<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17760/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15721090
--- Diff: bin/run-example ---
@@ -29,7 +29,9 @@ if [ -n "$1" ]; then
else
echo "Usage: ./bin/run-example <example-class> [example-args]" 1>&2
echo " - set MASTER=XX to use a specific master" 1>&2
- echo " - can use abbreviated example class name (e.g. SparkPi, mllib.LinearRegression)" 1>&2
+ echo " - can use abbreviated example class name relative to com.apache.spark.examples" 1>&2
+ echo " (e.g. SparkPi, mllib.LinearRegression, streaming.KinesisWordCountASL)" 1>&2
+ echo " - to run the Kinesis Spark Streaming example, make sure you build with -Pkinesis-asl" 1>&2
--- End diff --
@pwendell Is it a good idea to have this in the main bin/run-example?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15564230
--- Diff: extras/spark-kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.util.List
+import scala.collection.JavaConversions.asScalaBuffer
+import scala.collection.mutable.ArrayBuffer
+import org.apache.spark.Logging
+import org.apache.spark.streaming.util.ManualClock
+import org.apache.spark.streaming.util.SystemClock
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor
+import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer
+import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownReason
+import com.amazonaws.services.kinesis.model.Record
+import scala.compat.Platform
+import org.apache.spark.streaming.util.Clock
+
+/**
+ * Kinesis-specific implementation of the Kinesis Client Library (KCL) IRecordProcessor.
+ * This implementation operates on the Array[Byte] from the KinesisReceiver.
+ * The Kinesis Worker creates an instance of this KinesisRecordProcessor upon startup.
+ *
+ * @param Kinesis receiver
+ * @param workerId for logging purposes
+ * @param checkpoint state
+ */
+private[streaming] class KinesisRecordProcessor(
+ receiver: KinesisReceiver,
+ workerId: String,
+ checkpointState: CheckpointState) extends IRecordProcessor with Logging {
+
+ /** shardId to be populated during initialize() */
+ var shardId: String = _
+
+ /**
+ * The Kinesis Client Library calls this method during IRecordProcessor initialization.
+ *
+ * @param shardId assigned by the KCL to this particular RecordProcessor.
+ */
+ override def initialize(shardId: String) {
+ logInfo(s"Initialize: Initializing workerId $workerId with shardId $shardId")
+
--- End diff --
extra space.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1434#discussion_r15631461
--- Diff: extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
+import org.apache.spark.streaming.api.java.JavaStreamingContext
+import org.apache.spark.streaming.dstream.ReceiverInputDStream
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+
+
+/**
+ * Facade to create the Scala-based or Java-based streams.
+ * Also, contains a reusable utility methods.
+ * :: Experimental ::
+ */
+@Experimental
+object KinesisUtils extends Logging {
+ /**
+ * Create an InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param StreamingContext object
+ * @param appName Kinesis Application Name. Kinesis Apps are mapped to Kinesis Streams
+ * by the Kinesis Client Library. If you change the App name or Stream name,
+ * the KCL will throw errors.
+ * @param stream Kinesis Stream Name
+ * @param endpoint url of Kinesis service
+ * @param checkpoint interval (millis) for Kinesis checkpointing (not Spark checkpointing).
+ * See the Kinesis Spark Streaming documentation for more details on the different types
+ * of checkpoints.
+ * @param initialPositionInStream in the absence of Kinesis checkpoint info, this is the
+ * worker's initial starting position in the stream.
+ * The values are either the beginning of the stream per Kinesis' limit of 24 hours
+ * (InitialPositionInStream.TRIM_HORIZON) or the tip of the stream
+ * (InitialPositionInStream.LATEST).
+ * The default is TRIM_HORIZON to avoid potential data loss. However, this presents the risk
+ * of processing records more than once.
+ * @param storageLevel The default is StorageLevel.MEMORY_AND_DISK_2 which replicates in-memory
+ * and on-disk to 2 nodes total (primary and secondary)
+ *
+ * @return ReceiverInputDStream[Array[Byte]]
+ */
+ def createStream(
+ ssc: StreamingContext,
+ appName: String,
+ stream: String,
+ endpoint: String,
+ checkpointIntervalMillis: Long,
+ initialPositionInStream: InitialPositionInStream,
+ storageLevel: StorageLevel): ReceiverInputDStream[Array[Byte]] = {
+ ssc.receiverStream(new KinesisReceiver(appName, stream, endpoint, checkpointIntervalMillis,
+ initialPositionInStream, storageLevel))
+ }
+
+ /**
+ * Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
+ *
+ * @param JavaStreamingContext object
+ * @param appName Kinesis Application Name. Kinesis Apps are mapped to Kinesis Streams
+ * by the Kinesis Client Library. If you change the App name or Stream name,
+ * the KCL will throw errors.
+ * @param stream Kinesis Stream Name
+ * @param endpoint url of Kinesis service
--- End diff --
Can you give couple of examples of endpoints to give an idea what they look like
`@param endpoint Url of the Kinesis service (e.g. https://kinesis.us-west-2.amazon.com.or.something.like.that)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50250292
Chris, is this new code from scratch, or is it based on Parviz's old pull request?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1434#issuecomment-50952682
QA results for PR 1434:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>public final class JavaKinesisWordCountASL {<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17736/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---