You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "mxm (via GitHub)" <gi...@apache.org> on 2023/05/15 11:03:56 UTC
[GitHub] [flink-kubernetes-operator] mxm opened a new pull request, #597: [FLINK-32100] Use the total number of Kafka partitions as the max source parallelism
mxm opened a new pull request, #597:
URL: https://github.com/apache/flink-kubernetes-operator/pull/597
So far, we've taken the max number partitions we can find. However, the correct way to calculate the max source parallelism would be to sum the number of partitions of all topis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on pull request #597: [FLINK-32100] Use the total number of Kafka partitions as the max source parallelism
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on PR #597:
URL: https://github.com/apache/flink-kubernetes-operator/pull/597#issuecomment-1549875653
My only concern here is that we might get into a situation where the sum(partitions) does not have many divisors and will trigger a big scaleup of the source.
I wonder if we should make this logic configurable. In any case we should at least add some note/warning about this to the docs and maybe suggest setting an explicit max parallelism on the source operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mxm commented on pull request #597: [FLINK-32100] Use the total number of Kafka partitions as the max source parallelism
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #597:
URL: https://github.com/apache/flink-kubernetes-operator/pull/597#issuecomment-1550041077
I agree, probably makes sense to disable key group alignment for sources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mxm commented on pull request #597: [FLINK-32100] Use the total number of Kafka partitions as the max source parallelism
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #597:
URL: https://github.com/apache/flink-kubernetes-operator/pull/597#issuecomment-1550025309
For reference: https://github.com/apache/flink-kubernetes-operator/blob/80b13b52183e8a96c8a7de3fcca97b60f415a3a2/flink-kubernetes-operator-autoscaler/src/main/java/org/apache/flink/kubernetes/operator/autoscaler/JobVertexScaler.java#L272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mxm commented on pull request #597: [FLINK-32100] Use the total number of Kafka partitions as the max source parallelism
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #597:
URL: https://github.com/apache/flink-kubernetes-operator/pull/597#issuecomment-1550024389
We don't know how many divisors the number of partitions of a topic has. Similarly, we don't know how how many divisors the sum of the number of partitions of all topics has. What you say applies to any operator, not only to source operators. Source operators are limited by the max partitions which are usually smaller than the configured max vertex which will always be the max also for all vertices including sources.
From my experience the worst thing that can happen is that the source is underprovisioned which this PR tries to avoid. Underprovisioned sources suffer from watermark alignment issues and will build huge lag. We need to allow sources to reach the max scale up which is the sum of all partitions.
Let's say you have 73 (prime number) partitions because you read from topic1 with 60 partitions and topic2 with 13 partitions. The autoscaler can now decide to scale between 1 and 73. Any of these parallelisms will work. In case of a non-prime sum of partitions, the scale up will at worst case be `Math.min(sum_of_partitions / 2, max_parallelism)` in case of key-group alignment. Otherwise, we will choose the target parallelism like in the prime example.
I think we can think about adding a config option to disable keygroup alignment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mxm merged pull request #597: [FLINK-32100] Use the total number of Kafka partitions as the max source parallelism
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm merged PR #597:
URL: https://github.com/apache/flink-kubernetes-operator/pull/597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org