[jira] [Commented] (KAFKA-3979) Optimize memory used by replication process by using adaptive fetch message size

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385865#comment-15385865 ] 

ASF GitHub Bot commented on KAFKA-3979:
---------------------------------------

GitHub user nepal opened a pull request:

    https://github.com/apache/kafka/pull/1642

    [KAFKA-3979] Optimize memory used by replication process by using ada…

    …ptive fetch message size

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nepal/kafka adaptive-replica-fetch-size

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/1642.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1642
    
----
commit 300be1adb0089df4990c0bc2f8aaf0a50556fc8b
Author: Andrey L. Neporada <an...@yandex-team.ru>
Date:   2016-07-20T13:37:15Z

    [KAFKA-3979] Optimize memory used by replication process by using adaptive fetch message size

----


> Optimize memory used by replication process by using adaptive fetch message size
> --------------------------------------------------------------------------------
>
>                 Key: KAFKA-3979
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3979
>             Project: Kafka
>          Issue Type: Improvement
>          Components: replication
>    Affects Versions: 0.10.0.0
>            Reporter: Andrey Neporada
>
> Current replication process fetches messages in replica.fetch.max.bytes-sized chunks.
> Since replica.fetch.max.bytes should be bigger than max.message.bytes for replication to work, one can face big memory consumption for replication process, especially for installations with big number of partitions.
> Proposed solution is to try to fetch messages in smaller chunks (say replica.fetch.base.bytes).
> If we encounter message bigger than current fetch chunk, we increase chunk (f.e. twofold) and retry. After replicating this bigger message, we shrunk fetch chunk size back until it reaches replica.fetch.base.bytes
> replica.fetch.base.bytes should be chosen big enough not to affect throughput and to be bigger than most of messages.
> However, it can be much less than replica.fetch.max.bytes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)