[jira] [Created] (SPARK-14693) Spark Streaming Context Hangs on Start

["Evan Oman (JIRA)" <ji...@apache.org>]

Evan Oman created SPARK-14693:
---------------------------------

             Summary: Spark Streaming Context Hangs on Start
                 Key: SPARK-14693
                 URL: https://issues.apache.org/jira/browse/SPARK-14693
             Project: Spark
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 1.6.1, 1.6.0
         Environment: Databricks Cloud
            Reporter: Evan Oman


All,

I am trying to use Kinesis with Spark Streaming on Spark 1.6.0 via Databricks and my `ssc.start()` command is hanging. 

I am using the following function to make my Spark Streaming Context:

{code:borderStyle=solid}
def creatingFunc(sc: SparkContext): StreamingContext = 
{
		// Create a StreamingContext
		val ssc = new StreamingContext(sc, Seconds(batchIntervalSeconds))

		// Creata a Kinesis stream
		val kinesisStream = KinesisUtils.createStream(ssc,
			kinesisAppName, kinesisStreamName,
			kinesisEndpointUrl, RegionUtils.getRegionByEndpoint(kinesisEndpointUrl).getName,
			InitialPositionInStream.LATEST, Seconds(kinesisCheckpointIntervalSeconds),
			StorageLevel.MEMORY_AND_DISK_SER_2, config.awsAccessKeyId, config.awsSecretKey)

		kinesisStream.print()

		ssc.remember(Minutes(1))
		ssc.checkpoint(checkpointDir)
		ssc
}
{code}


However when I run the following to start the streaming context:

{code:borderStyle=solid}
// Stop any existing StreamingContext 
val stopActiveContext = true
if (stopActiveContext) {	
  StreamingContext.getActive.foreach { _.stop(stopSparkContext = false) }
} 

// Get or create a streaming context.
val ssc = StreamingContext.getActiveOrCreate(() => main.creatingFunc(sc))

// This starts the streaming context in the background. 
ssc.start()
{code}

The last bit, `ssc.start()`, hangs indefinitely without issuing any log messages. I am running this on a freshly spun up cluster with no other notebooks attached so there aren't any other streaming contexts running.

Any thoughts?

Additionally, here are the libraries I am using (from my build.sbt file):

{code:borderStyle=solid}
"org.apache.spark" % "spark-core_2.10" % "1.6.0"
"org.apache.spark" % "spark-sql_2.10" % "1.6.0"
"org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.6.0"
"org.apache.spark" % "spark-streaming_2.10" % "1.6.0"
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org