[jira] [Commented] (KAFKA-1417) Very slow initial high-level consumer startup in low traffic/blocking fetch scenario

["Sam Meder (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979211#comment-13979211 ] 

Sam Meder commented on KAFKA-1417:
----------------------------------

The simple, although not the most efficient solution would be to add another simple consumer instance in each fetcher...

> Very slow initial high-level consumer startup in low traffic/blocking fetch scenario
> ------------------------------------------------------------------------------------
>
>                 Key: KAFKA-1417
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1417
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.8.0
>            Reporter: Sam Meder
>            Assignee: Neha Narkhede
>
> We're seeing very slow startup times when starting a high level consumer in a low traffic/blocking fetch type setup. The example we've come across has a consumer that is set up to use 3 topics and uses a 20s/1 byte fetch timeout. What happens is that the leader finder thread adds partitions one by one and since the offset is not know this causes a call to figure out the offset. This call uses the fetcher threads simple consumer instance and locks around the call. Initially this is not a problem, but as soon as the fetcher thread has some partitions it will start fetching and since this is a low traffic situation the fetch will at least sometimes take up to 20s (again locking around the simple consumer). This leads to behavior like:
> # Finder thread adds a partition
> # Data thread notices it has partitions to fetch data for, locks the consumer for 20s
> # Finder thread tries to add a partition, tries to lock consumer and blocks for 20s
> # Rinse, repeat for each partition



--
This message was sent by Atlassian JIRA
(v6.2#6252)