Re: SSL between Kafka and Spark Streaming API

[SURESH CHAGANTI <ch...@gmail.com>]

Hi Everyone, 

I am trying to enable the SSL between my Apache Spark 1.4.1 and Kafka 0.9.0.0 and I am using spark-streaming-kafka_2.10 Jar to connect to Kafka and I am using KafkaUtils.createDirectStream method to read the data from Kafka topic. Initially, I got OOM issue and I have resolved it by increasing the Driver memory, after that I am seeing below issue, I have done little bit of reading and found out that spark-streaming-kafka_2.10 uses Kafka 0.8.2.1 API, which doesn't support SSL (Kafka supports SSL only after 0.9.0.0 versions). are there any alternatives to enable SSL between Spark 1.4.1 and Kafka 0.9.0.0, can someone throw some lite on this pls ? here are few links I have referred 

http://grokbase.com/p/kafka/users/158wy0wtxk/ssl-between-kafka-and-spark-streaming-api <http://grokbase.com/p/kafka/users/158wy0wtxk/ssl-between-kafka-and-spark-streaming-api>
SSL between Kafka and Spark <http://stackoverflow.com/questions/37743490/ssl-between-kafka-and-spark>
http://spark.apache.org/docs/latest/streaming-kafka-integration.html <http://spark.apache.org/docs/latest/streaming-kafka-integration.html>
https://issues.apache.org/jira/browse/SPARK-15089 <https://issues.apache.org/jira/browse/SPARK-15089>
+++++++++++++++++++++++++++++++++ Here is the log

iableProperties: Property security.protocol is not valid
16/10/24 18:25:09 WARN utils.VerifiableProperties: Property ssl.truststore.location is not valid
16/10/24 18:25:09 WARN utils.VerifiableProperties: Property ssl.truststore.password is not valid
16/10/24 18:25:09 INFO utils.VerifiableProperties: Property zookeeper.connect is overridden to 
16/10/24 18:25:09 INFO consumer.SimpleConsumer: Reconnect due to error
Exception in thread "main" org.apache.spark.SparkException: java.io.EOFException
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:99)

 @
Harsha <http://grokbase.com/user/Harsha/YJque1PXQTKLA57prYzPCR>, Cody  any thought ?