You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/05/03 23:35:09 UTC
[GitHub] [kafka] ijuma opened a new pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
ijuma opened a new pull request #8607:
URL: https://github.com/apache/kafka/pull/8607
In the case described in the JIRA, there was a 50%+ increase in the total fetch request
rate in 2.4.0 due to this change.
I included a few additional clean-ups:
* Simplify `findPreferredReadReplica` and avoid unnecessary collection copies.
* Use `LongSupplier` instead of `Supplier<Long>` in `SubscriptionState` to
avoid unnecessary boxing.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
hachikuji commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-623630601
Filed this issue: https://issues.apache.org/jira/browse/KAFKA-9952.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
hachikuji commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-623617879
@ijuma Thanks for the patch. I think the change makes sense. Part of me is inclined to drop the logic for immediate propagation of high watermark entirely. Instead, we could say propagation is bounded by the max wait time which users could potentially tune for their own needs. Otherwise enabling this feature becomes sort of risky. We just need to be a little careful to ensure that the partition still gets included in the fetch response if there is a high watermark update. I'll open a JIRA and we can discuss this separately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-623787840
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-623240751
1 job passed, 1 failed due to a spotBugs issue with Scala 2.12 and one failed with an unrelated flaky test failure:
> org.apache.kafka.streams.integration.GlobalKTableIntegrationTest.shouldKStreamGlobalKTableLeftJoin
I pushed a fix for the spotBugs issue and will try to add a test verifying this behavior since we don't see to have any.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-623470067
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jlisam commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
jlisam commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-662649186
hello @ijuma, I was wondering if you knew when this fix would be backported. We are currently on 2.5.0 and we are currently experiencing this issue. Thank you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-662685160
@jlisam It's too late to be included in 2.5.1 (currently in RC stage), but it will be part of 2.6.0 (also at RC stage). Would you be able to upgrade to 2.6.0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-623620280
@hachikuji I was thinking the exact same thing. :) I thought this PR could be the simple one we can backport and then we can discuss the larger change separately as you suggested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-623847180
1 job passed, the other 2 had unrelated flaky failures:
> kafka.integration.MetricsDuringTopicCreationDeletionTest.testMetricsDuringTopicCreateDelete
> org.apache.kafka.streams.integration.GlobalKTableIntegrationTest.shouldKStreamGlobalKTableJoin
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldAllowConcurrentAccesses
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] gwenshap commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
gwenshap commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-623848663
The new test looks good and validates the change made in this PR.
LGTM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-623757017
I added a test. @mumrah @gwenshap maybe one of you can take a quick look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji edited a comment on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
hachikuji edited a comment on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-623617879
@ijuma Thanks for the patch. I think the change makes sense. Part of me is inclined to drop the logic for the immediate propagation of the high watermark entirely. Instead, we could say propagation is bounded by the max wait time which users could potentially tune for their own needs. Otherwise enabling this feature becomes sort of risky. We just need to be a little careful to ensure that the partition still gets included in the fetch response if there is a high watermark update. I'll open a JIRA and we can discuss this separately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jlisam commented on pull request #8607: KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
Posted by GitBox <gi...@apache.org>.
jlisam commented on pull request #8607:
URL: https://github.com/apache/kafka/pull/8607#issuecomment-662695001
@ijuma thanks for the prompt response. I think we should be able to upgrade to 2.6.0 but ideally a backport would be better given that we have spent some time validating the 2.5.0 cluster performance. Anyhow, thanks for fixing this 😄 !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org