You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/08/27 09:02:00 UTC

[jira] [Updated] (FLINK-40445) Autoscaler: integration test for Kafka/Pulsar source partition-metric detection

     [ https://issues.apache.org/jira/browse/FLINK-40445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated FLINK-40445:
-----------------------------------
    Labels: pull-request-available  (was: )

> Autoscaler: integration test for Kafka/Pulsar source partition-metric detection
> -------------------------------------------------------------------------------
>
>                 Key: FLINK-40445
>                 URL: https://issues.apache.org/jira/browse/FLINK-40445
>             Project: Flink
>          Issue Type: Bug
>          Components: Autoscaler, Kubernetes Operator
>            Reporter: Purushottam Sinha
>            Priority: Minor
>              Labels: pull-request-available
>
> Problem
> ScalingMetricCollector derives a source's partition count by parsing Kafka/Pulsar metric names (PartitionMetricNameParser, added in FLINK-40443 / PR #1187). This path has no automated coverage against a real broker, so a regression in partition detection could go unnoticed until it affects source autoscaling.
> Evidence
> - flink-autoscaler/.../ScalingMetricCollector.java → updateKafkaPulsarSourceNumPartitions uses PartitionMetricNameParser; only unit/differential tests exist today.
> - FLINK-40443 was verified manually against Apache Kafka 3.9.1 and Pulsar 3.3.1 on minikube, not in CI.
> - Existing precedent: flink-autoscaler-plugin-jdbc uses Testcontainers for the JDBC state store.
> Proposed fix
> - Add a Testcontainers-based integration test (separate PR; not blocking 1.16.0) running a source job against real Kafka and Pulsar.
> - Assert derived numSourcePartitions for: multi-topic summing, dotted/hyphen topic names, source parallelism > 1, and multiple source vertices in one job.
> Acceptance
> - Test asserts detected partition count equals broker ground truth for each dimension above.
> - Runs in CI.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)