You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Maximilian Michels (Jira)" <ji...@apache.org> on 2023/12/07 15:32:00 UTC

[jira] [Assigned] (FLINK-31400) Add autoscaler integration for Iceberg source

     [ https://issues.apache.org/jira/browse/FLINK-31400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Maximilian Michels reassigned FLINK-31400:
------------------------------------------

    Assignee: Mason Chen

> Add autoscaler integration for Iceberg source
> ---------------------------------------------
>
>                 Key: FLINK-31400
>                 URL: https://issues.apache.org/jira/browse/FLINK-31400
>             Project: Flink
>          Issue Type: New Feature
>          Components: Autoscaler, Kubernetes Operator
>            Reporter: Maximilian Michels
>            Assignee: Mason Chen
>            Priority: Major
>
> A very critical part in the scaling algorithm is setting the source processing rate correctly such that the Flink pipeline can keep up with the ingestion rate. The autoscaler does that by looking at the {{pendingRecords}} Flink source metric. Even if that metric is not available, the source can still be sized according to the busyTimeMsPerSecond metric, but there will be no backlog information available. For Kafka, the autoscaler also determines the number of partitions to avoid scaling higher than the maximum number of partitions.
> In order to support a wider range of use cases, we should investigate an integration with the Iceberg source. As far as I know, it does not expose the pedingRecords metric, nor does the autoscaler know about other constraints, e.g. the maximum number of open files.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)