About Kafka08Fetcher and Kafka010Fetcher

[Jaxon Hu <hu...@gmail.com>]

In Kafka08Fetcher, it use  Map<Node,SimpleConsumerThread> to manage
multi-threads. But I notice in Kafka09Fetcher or Kafka010Fetcher, it's
gone. So how Kafka09Fetcher implements multi-threads read partitions from
kafka?

Re: About Kafka08Fetcher and Kafka010Fetcher

["Tzu-Li (Gordon) Tai" <tz...@apache.org>]

Hi Jaxon,

The threading model is implemented differently between the Kafka08Fetcher and all other fetcher versions higher than 0.9+ because the Kafka Java clients used between these versions have different abstraction levels.

The Kafka08Fetcher still uses the low-level `SimpleConsumer` API, which only allows connection to a single Kafka broker. Therefore, the Kafka08Fetcher maintains a thread for each broker that it connects to. If a subtask’s assigned partitions all exist on the same Kafka broker, only one thread will be created.

On the other hand, in versions 0.9+, the high-level `KafkaConsumer` API is used, which hides away the complexity for per-broker connections. I believe that the `KafkaConsumer` client still creates per-broker threads internally, if its assigned partitions span multiple brokers.

Cheers,
Gordon

On 2 January 2018 at 8:22:22 AM, Timo Walther (twalthr@apache.org) wrote:

Maybe Gordon (in CC) can answer your question.  


Am 1/1/18 um 3:36 PM schrieb Jaxon Hu:  
> In Kafka08Fetcher, it use  Map<Node,SimpleConsumerThread> to manage  
> multi-threads. But I notice in Kafka09Fetcher or Kafka010Fetcher, it's  
> gone. So how Kafka09Fetcher implements multi-threads read partitions  
> from kafka?  

Re: About Kafka08Fetcher and Kafka010Fetcher

[Timo Walther <tw...@apache.org>]

Maybe Gordon (in CC) can answer your question.


Am 1/1/18 um 3:36 PM schrieb Jaxon Hu:
> In Kafka08Fetcher, it use  Map<Node,SimpleConsumerThread> to manage 
> multi-threads. But I notice in Kafka09Fetcher or Kafka010Fetcher, it's 
> gone. So how Kafka09Fetcher implements multi-threads read partitions 
> from kafka?