You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "Manjunath, Kiran" <ki...@akamai.com> on 2016/10/10 09:17:02 UTC

Spark Streaming Custom Receivers - How to use metadata store API during processing

Hello,

This is a reposting of the question which is available in stackOverflow (posted by another user).
http://stackoverflow.com/questions/35271270/how-to-access-metadata-stored-by-spark-streaming-custom-receiver

Question:
I need to store meta information (basically certain properties of the record) in the custom receiver and then use the same information in the receiverStream caller.
Can anyone help me with a sample code on how to use it. (Currently I am able to process the rdd without the meta information).

I did search the GitHub and found one usage in KinesisReceiver.scala.
However, it did go over my head in understanding the code and usage.

https://github.com/apache/spark/blob/39e2bad6a866d27c3ca594d15e574a1da3ee84cc/external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala#L282


Any help is appreciated.

Thanks!
Regards,
Kiran