You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "hudeqi (via GitHub)" <gi...@apache.org> on 2023/06/14 11:28:30 UTC
[GitHub] [kafka] hudeqi opened a new pull request, #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
hudeqi opened a new pull request, #13852:
URL: https://github.com/apache/kafka/pull/13852
### Activation
As the config 'segment.bytes' for topics related MM2(such as offset.storage.topic, config.storage.topic,status.storage.topic), if following the default configuration of the broker or set it larger, then when the MM cluster runs many and complicated tasks, especially the log volume of the topic 'offset.storage.topic' is very large, it will affect the restart speed of the MM workers.
After investigation, the reason is that a consumer needs to be started to read the data of ‘offset.storage.topic’ at startup. Although this topic is set to compact, if the 'segment size' is set to a large value, such as the default value of 1G, then this topic may have tens of gigabytes of data that cannot be compacted and has to be read from the earliest (because the active segment cannot be cleaned), which will consume a lot of time (in our online environment, we found that this topic stores 13G of data, it took nearly half an hour for all the data to be consumed), which caused the worker to be unable to start and execute tasks for a long time.
### Solution
The number of consumer threads can also be adjusted, but I think it may be easier to set a upper limit for the 'segment size', for example, refer to the default value of __consumer_offsets: 100MB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13852:
URL: https://github.com/apache/kafka/pull/13852#issuecomment-1608769851
> @hudeqi sorry, this is a tricky issue and I'm trying to take time to think things through :)
>
> I hate to say it, but I don't think we can make this change or anything like it without a KIP. This is for two reasons:
>
> 1. We're effectively changing the default value for the `offset.storage.topic.segment.bytes` property (even if we don't implement this change with that exact logic), which counts as a change to public API for the project
> 2. By explicitly setting a value for the offset topic's `segment.bytes` property, we cause any broker-side value for the [log.segment.bytes property](https://kafka.apache.org/documentation.html#brokerconfigs_log.segment.bytes) to be ignored. If the broker uses a lower value for this property than our default, then we may make things worse instead of better
>
> I still think it's likely that decreasing the segment size for the offsets topic would help, but it'd be nice if we could get the kind of review that a KIP requires before making that kind of change.
>
> As far as increasing the number of consumer threads goes, I think it's really a question of what the performance bottleneck is when reading to the end of the topic. If CPU is the issue, then sure, it'd probably help to scale up the number of consumers. However, if network transfer between the worker and the Kafka cluster is the limiting factor, then it won't have any impact. The nice thing about decreasing the segment size is that (as long as it leads to a reduction in the total size of the offsets topic), it would help in either case: you'd have less data to consume from Kafka, and also less data to process on your Connect worker.
>
> This almost certainly varies depending on the environment Kafka Connect and Kafka are run in, but my hunch is that your fix here would be more effective than scaling up the number of consumers. I'd be curious to see if we could get benchmark numbers on that front, though.
Thanks, I will issue a KIP and reopen this PR later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] C0urante commented on pull request #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #13852:
URL: https://github.com/apache/kafka/pull/13852#issuecomment-1607920082
@hudeqi sorry, this is a tricky issue and I'm trying to take time to think things through :)
I hate to say it, but I don't think we can make this change or anything like it without a KIP. This is for two reasons:
1. We're effectively changing the default value for the `offset.storage.topic.segment.bytes` property (even if we don't implement this change with that exact logic), which counts as a change to public API for the project
2. By explicitly setting a value for the offset topic's `segment.bytes` property, we cause any broker-side value for the [log.segment.bytes property](https://kafka.apache.org/documentation.html#brokerconfigs_log.segment.bytes) to be ignored. If the broker uses a lower value for this property than our default, then we may make things worse instead of better
I still think it's likely that decreasing the segment size for the offsets topic would help, but it'd be nice if we could get the kind of review that a KIP requires before making that kind of change.
As far as increasing the number of consumer threads goes, I think it's really a question of what the performance bottleneck is when reading to the end of the topic. If CPU is the issue, then sure, it'd probably help to scale up the number of consumers. However, if network transfer between the worker and the Kafka cluster is the limiting factor, then it won't have any impact. The nice thing about decreasing the segment size is that (as long as it leads to a reduction in the total size of the offsets topic), it would help in either case: you'd have less data to consume from Kafka, and also less data to process on your Connect worker.
This almost certainly varies depending on the environment Kafka Connect and Kafka are run in, but my hunch is that your fix here would be more effective than scaling up the number of consumers. I'd be curious to see if we could get benchmark numbers on that front, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13852:
URL: https://github.com/apache/kafka/pull/13852#issuecomment-1600661298
Hi, could you take the time to look at this issue? I think it is meaningful. @C0urante @mimaison
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13852:
URL: https://github.com/apache/kafka/pull/13852#issuecomment-1591013378
Hello, please help to review this issue when you are free, thank you! @C0urante
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on pull request #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #13852:
URL: https://github.com/apache/kafka/pull/13852#issuecomment-1593109309
I am not familiar with MM2 @hudeqi. I will leave it up to the experts such as @C0urante and @mimaison to look into this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] C0urante commented on pull request #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #13852:
URL: https://github.com/apache/kafka/pull/13852#issuecomment-1600935108
Thanks @hudeqi for addressing this issue--23-minute worker startup is definitely something we should try to avoid! That said, I have some thoughts about this change.
First, we should not override user-supplied values, even if they exceed what we believe is a reasonable value for the segment size of an internal topic.
Second, although the ticket and description deal exclusively with the internal offsets topic, it looks like the changes here also touch on the status and config topics. Have you verified that it's also necessary to change the segment size for those topics, and that the new value makes sense for them?
Third, I don't think we need to touch on the `TopicAdmin` class for this, and can instead leverage existing logic introduced with [KIP-605](https://cwiki.apache.org/confluence/display/KAFKA/KIP-605%3A+Expand+Connect+Worker+Internal+Topic+Settings). We can set the `offset.storage.topic.segment.bytes` property for `DistributedHerder` instances and let existing code handle the rest.
Fourth, it looks like this change affects not just MirrorMaker 2, but all of Kafka Connect. Is this intentional? It seems like this change would be beneficial for all source connectors and not just MM2, but if we want to play it safe and only alter the behavior of dedicated MM2 clusters, we can move the changes to the [MirrorMakerConfig class](https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMakerConfig.java) and leave Kafka Connect unchanged.
Finally, I'm curious about the value of this approach overall. It only affects newly-created Connect clusters; existing ones will not benefit. Plus, users can already configure the segment size (and nearly any other property) of the offsets topic. And if a 13G offsets topic leads to 23-minute startup time, can we be confident that reducing the segment size to 100MB will prevent that from happening? The offsets topic uses 25 partitions by default, so worst case we'd still see 25 * 100 * 2 (one active segment, one prior segment) = 5000MB = 5GB in the offsets topic, which would still take almost nine minutes to read through on worker startup. I understand that the size of the offsets topic is a pain point and I'd like to try to address it; I'm just wondering if this is really the best we can do here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13852:
URL: https://github.com/apache/kafka/pull/13852#issuecomment-1602409354
Thank you for your reply! @C0urante, seeing so many of your thoughts, I think it is very meaningful to think about! Below I will give my actual results and thoughts for each question.
1. I agree with you, if it is not mandatory to overwrite the user-defined value, then we may need to give a warning log to prompt for this case.
2. The reason why I encountered this kind of case is that there are too many topics synchronized in the MM cluster, too many partitions (very common and may not be avoided), the frequent update and storage of offset information leads to too large internal offset topic. As for internal config topic and internal status topic, I think it is difficult to meet the conditions for a large increase, and I haven't encountered it yet. Therefore, if we follow the "principle of least change", we may not need to make any adjustments for these two internal topics.
3. It's ok here if we respect the point of the first answer.
4. I've been calling it "MM2" probably inappropriately. Because in fact, I realized topic replication in the form of MirrorSourceConnector through DistributedHerder. Your idea is right, it may be beneficial for whole connect, I have no experience with other types of non-topic replication, this point needs your deciding. But if we use the connect cluster to achieve topic replication like me, I think this problem also needs to be solved.
5. This is how I do it as you say: if the current connect cluster already exists, I directly adjust the log segment size corresponding to the internal offset topic through “kafka-topic.sh”, which is a bit tricky. Although I mentioned setting 100MB in PR, in practice, I think it may still be a bit large. I set it to 50MB, and finally the startup time was shortened to 30s(It has been compacted to a total of only about 700MB), which may not have reached the 'worst case' you said (every partition It is full and the prior segment is also full)'. Maybe we can also increase the number of consumer threads that read offsets, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13852:
URL: https://github.com/apache/kafka/pull/13852#issuecomment-1606067982
> Thank you for your reply! @C0urante, seeing so many of your thoughts, I think it is very meaningful to think about! Below I will give my actual results and thoughts for each question.
>
> 1. I agree with you, if it is not mandatory to overwrite the user-defined value, then we may need to give a warning log to prompt for this case.
> 2. The reason why I encountered this kind of case is that there are too many topics synchronized in the MM cluster, too many partitions (very common and may not be avoided), the frequent update and storage of offset information leads to too large internal offset topic. As for internal config topic and internal status topic, I think it is difficult to meet the conditions for a large increase, and I haven't encountered it yet. Therefore, if we follow the "principle of least change", we may not need to make any adjustments for these two internal topics.
> 3. It's ok here if we respect the point of the first answer.
> 4. I've been calling it "MM2" probably inappropriately. Because in fact, I realized topic replication in the form of MirrorSourceConnector through DistributedHerder. Your idea is right, it may be beneficial for whole connect, I have no experience with other types of non-topic replication, this point needs your deciding. But if we use the connect cluster to achieve topic replication like me, I think this problem also needs to be solved.
> 5. This is how I do it as you say: if the current connect cluster already exists, I directly adjust the log segment size corresponding to the internal offset topic through “kafka-topic.sh”, which is a bit tricky. Although I mentioned setting 100MB in PR, in practice, I think it may still be a bit large. I set it to 50MB, and finally the startup time was shortened to 30s(It has been compacted to a total of only about 700MB), which may not have reached the 'worst case' you said (every partition It is full and the prior segment is also full)'. Maybe we can also increase the number of consumer threads that read offsets, what do you think?
Hi, @C0urante Did you see my reply? What are your thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics [kafka]
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi closed pull request #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
URL: https://github.com/apache/kafka/pull/13852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] github-actions[bot] commented on pull request #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13852:
URL: https://github.com/apache/kafka/pull/13852#issuecomment-1734772707
This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch) <p> If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13852: KAFKA-15086:Set a reasonable segment size upper limit for MM2 internal topics
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13852:
URL: https://github.com/apache/kafka/pull/13852#issuecomment-1593081298
Hello, please help to review this issue when you are free, thank you! @divijvaidya @mimaison
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org