[jira] [Updated] (KAFKA-170) Support for non-blocking polling on multiple streams

["Neha Narkhede (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neha Narkhede updated KAFKA-170:
--------------------------------

    Issue Type: Sub-task  (was: New Feature)
        Parent: KAFKA-364
    
> Support for non-blocking polling on multiple streams
> ----------------------------------------------------
>
>                 Key: KAFKA-170
>                 URL: https://issues.apache.org/jira/browse/KAFKA-170
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: core
>    Affects Versions: 0.8
>            Reporter: Jay Kreps
>            Priority: Critical
>              Labels: replication
>
> Currently we provide a blocking iterator in the consumer. This is a good mechanism for consuming data from a single topic, but is limited as a mechanism for polling multiple streams.
> For example if one wants to implement a non-blocking union across multiple streams this is hard to do because calls may block indefinitely. A similar situation arrises if trying to implement a streaming join of between two streams.
> I would propose two changes:
> 1. Implement a next(timeout) interface on KafkaMessageStream. This will easily handle some simple cases with minimal change. This handles certain limited cases nicely and is easy to implement, but doesn't actually cover the two cases above.
> 2. Add an interface to poll streams.
> I don't know the best approach for the later api, but it is important to get it right. One option would be to add a ConsumerConnector.drainTopics("topic1", "topic2", ...) which blocks until there is at least one message and then returns a list of triples (topic, partition, message).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira