[GitHub] [spark] dongjinleekr commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming

[GitBox <gi...@apache.org>]

dongjinleekr commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#issuecomment-513204523
 
 
   Here is the update, rebased against the latest master.
   
   @zsxwing About the Kafka client version, [the headers functionality is supported from 0.11.0.0](https://issues.apache.org/jira/browse/KAFKA-4208). As soon as this PR is merged, the note about the client version limitation should be added to the Kafka integration guide. I filed [a JIRA issue](https://issues.apache.org/jira/browse/SPARK-28446) about it.
   
   @srowen I reviewed `KafkaRelation#buildScan` to make the routine mapping `KafkaSourceRDD` to `InternalRow` as you proposed, but concluded that `KafkaUtils.toUnsafeMapData(cr.headers)` expression couldn't be added conditionally. Here is why: the `includeHeaders` flag is unavailable from the mapping function given to `RDD#map`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org