You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2015/12/10 04:09:11 UTC

[jira] [Commented] (KAFKA-2975) The newtorkClient should request a metadata update after it gets an error in the handleResponse()

    [ https://issues.apache.org/jira/browse/KAFKA-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049939#comment-15049939 ] 

ASF GitHub Bot commented on KAFKA-2975:
---------------------------------------

GitHub user MayureshGharat opened a pull request:

    https://github.com/apache/kafka/pull/658

    KAFKA-2975

    The newtorkClient should request a metadata update after it gets an error in the handleResponse().
    
    Currently in data pipeline,
    1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is set to 5 min
    2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries torefresh its Metadata.
    3) It gets LeaderNotAvailableException, may be because the topic is not created yet.
    4) Now its metadata does not have any information about that topic.
    5) It will wait for 5 min to do the next refresh.
    6) In the mean time the batches sitting in the accumulator will expire and the mirror makers die to avoid data loss.
    
    To overcome this we need to refresh the metadata after 3).
    
    Well there is an alternative solution to have the metadataExpiry set to be less then requestTimeout, but this will mean we make more metadataRequest over the wire in normal scenario as well.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/MayureshGharat/kafka kafka-2975

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/658.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #658
    
----
commit 8c7534b5ff89960e26db22a27e183d10002aeb01
Author: Mayuresh Gharat <mg...@mgharat-ld1.linkedin.biz>
Date:   2015-12-10T03:04:37Z

    The newtorkClient should request a metadata update after it gets an error in response

----


> The newtorkClient should request a metadata update after it gets an error in the handleResponse()
> -------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-2975
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2975
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>            Reporter: Mayuresh Gharat
>            Assignee: Mayuresh Gharat
>
> Currently in data pipeline, 
> 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is set to 5 min
> 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries torefresh its Metadata.
> 3) It gets LeaderNotAvailableException, may be because the topic is not created yet.
> 4) Now its metadata does not have any information about that topic.
> 5) It will wait for 5 min to do the next refresh.
> 6) In the mean time the batches sitting in the accumulator will expire and the mirror makers die to avoid data loss.
> To overcome this we need to refresh the metadata after 3).
> Well there is an alternative solution to have the metadataExpiry set to be less then requestTimeout, but this will mean we make more metadataRequest over the wire in normal scenario as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)