You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by salemi <al...@udo.edu> on 2017/12/09 18:07:20 UTC

Structured Streaming + Kafka 0.10. connectors + valueDecoder and messageHandler with python

Hi All,

we are currently using direct streams to get the data from a kafka topic as
followed

KafkaUtils.createDirectStream(ssc=self.streaming_context,
                                            topics=topics,
                                            kafkaParams=kafka_params,
                                            valueDecoder=message_decoder,
                                            messageHandler=message_handler)

We would like to switch to to Structured Streaming approach such as 

self.spark_session \
            .readStream \
            .format("kafka") \
            .option("kafka.bootstrap.servers", kafkaServers) \
            .option("subscribe", self.topic_id) \
            .option("auto.offset.reset", self.msgoffset)\
            .load()

I was wondering how I can apply the existing message_decoder and
message_handler  functions to the message stream?


Thank you,

Ali




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org