[jira] [Created] (FLINK-38381) Autoscaler does not enumerate partitions across kafka clusters

["Maximilian Michels (Jira)" <ji...@apache.org>]

Maximilian Michels created FLINK-38381:
------------------------------------------

             Summary: Autoscaler does not enumerate partitions across kafka clusters
                 Key: FLINK-38381
                 URL: https://issues.apache.org/jira/browse/FLINK-38381
             Project: Flink
          Issue Type: Bug
          Components: Autoscaler, Kubernetes Operator
    Affects Versions: kubernetes-operator-1.12.1, kubernetes-operator-1.12.0
            Reporter: Maximilian Michels
            Assignee: Maximilian Michels
             Fix For: kubernetes-operator-1.13.0


When using the DynamicKafkaSink, topics can be spread across multiple clusters. This used to work fine, but a regression has been added which considers partitions across different clusters to be identical. This limits the scale out of the source operator.

Here is an example:
{noformat}
"1.Source__Kafka_Source_(testTopic).kafkaCluster.my-cluster-1.KafkaSourceReader.topic.testTopic.partition.0.currentOffset",
"1.Source__Kafka_Source_(testTopic).kafkaCluster.my-cluster-2.KafkaSourceReader.topic.testTopic.partition.0.currentOffset"{noformat}
Those would result be treated as one partition, but there are two partitions from separate kafka clusters.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)