You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/04/26 12:42:29 UTC

[GitHub] [arrow-rs] alamb opened a new issue #148: Basic Kafka Reader

alamb opened a new issue #148:
URL: https://github.com/apache/arrow-rs/issues/148


   *Note*: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-10979
   
   Introduce a basic Kafka reader based on `rdkafka`. Exposes an `Iterator` interface which yields `Result<RecordBatch>`. 
   
   Columns in the batch are:
   
   *key* (Binary, nullable): The key of a message, if present.
   
   *payload* (Binary, nullable): The payload bytes of a message, if present.
   
   *topic* (Utf8): The topic of the message.
   
   *partition* (Int32): The partition of the message.
   
   *offset* (Int64): The offset of the message.
   
    
   
   Note that `rdkafka` has a C++ dependency (`librdkafka`), but we can choose to make this dynamically linked. `rdkafka` provides an `async` Consumer, but I have explicitly chosen the non-`async` Consumer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org