You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "HongXiang Ren (Jira)" <ji...@apache.org> on 2021/07/16 18:59:00 UTC
[jira] [Created] (BEAM-12631) Allow Inheritance from
KafkaUnboundedSource
HongXiang Ren created BEAM-12631:
------------------------------------
Summary: Allow Inheritance from KafkaUnboundedSource
Key: BEAM-12631
URL: https://issues.apache.org/jira/browse/BEAM-12631
Project: Beam
Issue Type: New Feature
Components: io-java-kafka
Reporter: HongXiang Ren
Reason:
We have dataflow to call one internal API (the API call take 60ms). So we want to start more threads to call this API. Kafka IO thread is control by split by default. And split is control by total partitions and system suggestion number. And system suggestion number is 4 * max workers. which is too low for us.
Solution:
Use facture to allow user inheritance KafkaUnboundedSource. and override the split function like
{code:java}
// code placeholder
@Override
public List<KafkaUnboundedSource<K, V>> split(int desiredNumSplits, PipelineOptions options)
throws Exception {
int split = this.getMaxSplits();
if(split <= 0){
return super.split(desiredNumSplits, options);
}else{
return super.split(split, options);
}
}
{code}
this may also useful for Override other feature of the KafkaUnboundedSource without copy all the io code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)