You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org> on 2018/05/02 11:23:00 UTC
[jira] [Comment Edited] (FLINK-9178) Add rate control for kafka
source
[ https://issues.apache.org/jira/browse/FLINK-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460891#comment-16460891 ]
Tzu-Li (Gordon) Tai edited comment on FLINK-9178 at 5/2/18 11:22 AM:
---------------------------------------------------------------------
[~app-tarush]
Sorry for slow late reply here.
Before we move on to an implementation of this inside the connector, have you looked into Kafka's own client configurations and whether or not they support this?
IIRC, there are configurations that allow controlling how much data is returned on each fetch, e.g. {{max.partition.fetch.bytes}}. Wouldn't this work?
was (Author: tzulitai):
[~app-tarush]
Sorry for slow late reply here.
Before we move on to an implementation of this inside the connector, have you looked into Kafka's own client configurations and whether or not they support this?
IIRC, there are configurations that allow controlling how much data is returned on each fetch, i.e. `max.partition.fetch.bytes`. Wouldn't this work?
> Add rate control for kafka source
> ---------------------------------
>
> Key: FLINK-9178
> URL: https://issues.apache.org/jira/browse/FLINK-9178
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
> Reporter: buptljy
> Assignee: Tarush Grover
> Priority: Major
>
> When I want to run the flink program from the earliest offset in Kafka, it'll be very easy to cause OOM if there are too much data, because of too many HeapMemorySegment in NetworkBufferPool.
> Maybe we should have some settings to control the rate of the receiving data?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)