You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ericzhifengchen (via GitHub)" <gi...@apache.org> on 2024/02/14 22:13:41 UTC
[PR] KAFKA-16259 Immutable MetadataCache to improve client performance [kafka]
ericzhifengchen opened a new pull request, #15376:
URL: https://github.com/apache/kafka/pull/15376
*More detailed description of your change,
current MetadataCache is partially (but not fully) immutable, thus read/write requires synchronization and led to high produce latency with large number of topics in a kafka cluster
This change improves performance of metadata cache read operation of native kafka producer with copy-on-write strategy
What is copy-on-write strategy
It’s a solution to reduce synchronized lock contention by making the object immutable, and always create a new instance when updating, but since the object is immutable, read operation will be free from waiting, thus produce latency reduced significantly
Besides performance, it can also make the metadata cache immutable from unexpected modification, reduce occurrence of code bugs due to incorrect synchronization
*Summary of testing strategy (including rationale)
Add unit test to cover concurrent read/write
Run in production, with large scale for over 3months
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16259 Immutable MetadataCache to improve client performance [kafka]
Posted by "msn-tldr (via GitHub)" <gi...@apache.org>.
msn-tldr commented on PR #15376:
URL: https://github.com/apache/kafka/pull/15376#issuecomment-1956540126
@ericzhifengchen The email you shared is anonymised with "*", I see it as `***@***.***`. Is that because replied using email & github automatically anonymised it 🤔
`42991652+msn-tldr@users.noreply.github.com`
Can you copy-paste email-id directly in a comment on the github web-page for this PR?
https://github.com/apache/kafka/pull/15376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16259 Immutable MetadataCache to improve client performance [kafka]
Posted by "ericzhifengchen (via GitHub)" <gi...@apache.org>.
ericzhifengchen commented on PR #15376:
URL: https://github.com/apache/kafka/pull/15376#issuecomment-1947427960
@ijuma Thanks for the info. confirmed it's done in [#15323](https://github.com/apache/kafka/pull/15323), I will close this one as duplicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16259 Immutable MetadataCache to improve client performance [kafka]
Posted by "msn-tldr (via GitHub)" <gi...@apache.org>.
msn-tldr commented on PR #15376:
URL: https://github.com/apache/kafka/pull/15376#issuecomment-1952339142
@ericzhifengchen It seems you had a similar idea on creating immutable metadata cache on the client to improve latency :)
I have created a follow-up https://github.com/apache/kafka/pull/15385 to add similar test to `testConcurrentUpdateAndGetCluster` in this PR. I can add you as a co-author to PR 15385, can you share the email with your github account? See steps on getting this information [here](https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors#required-co-author-information)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16259 Immutable MetadataCache to improve client performance [kafka]
Posted by "ericzhifengchen (via GitHub)" <gi...@apache.org>.
ericzhifengchen commented on PR #15376:
URL: https://github.com/apache/kafka/pull/15376#issuecomment-1955880312
Hi Mayank,
My github account is ***@***.***
Thanks,
Zhifeng
On Mon, Feb 19, 2024 at 4:22 AM Mayank Shekhar Narula <
***@***.***> wrote:
> @ericzhifengchen <https://github.com/ericzhifengchen> It seems you had a
> similar idea on creating immutable metadata cache on the client to improve
> latency :)
>
> I have created a follow-up #15385
> <https://github.com/apache/kafka/pull/15385> to add similar test to
> testConcurrentUpdateAndGetCluster in this PR. I can add you as a
> co-author to PR 15385, can you share the email with your github account?
> See steps on getting this information here
> <https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors#required-co-author-information>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/kafka/pull/15376#issuecomment-1952339142>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AQFQT7NDMG32VY42GWDJIPDYUM7ZZAVCNFSM6AAAAABDJGSNEGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJSGMZTSMJUGI>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16259 Immutable MetadataCache to improve client performance [kafka]
Posted by "msn-tldr (via GitHub)" <gi...@apache.org>.
msn-tldr commented on PR #15376:
URL: https://github.com/apache/kafka/pull/15376#issuecomment-1959270699
Added you as a co-author
https://github.com/apache/kafka/pull/15385/commits/e4f899714d36b53ec670e914243454ba478c67d4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16259 Immutable MetadataCache to improve client performance [kafka]
Posted by "ericzhifengchen (via GitHub)" <gi...@apache.org>.
ericzhifengchen commented on PR #15376:
URL: https://github.com/apache/kafka/pull/15376#issuecomment-1958593101
my github account is ericzhifengchen@gmail.com
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16259 Immutable MetadataCache to improve client performance [kafka]
Posted by "ericzhifengchen (via GitHub)" <gi...@apache.org>.
ericzhifengchen closed pull request #15376: KAFKA-16259 Immutable MetadataCache to improve client performance
URL: https://github.com/apache/kafka/pull/15376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16259 Immutable MetadataCache to improve client performance [kafka]
Posted by "ericzhifengchen (via GitHub)" <gi...@apache.org>.
ericzhifengchen commented on PR #15376:
URL: https://github.com/apache/kafka/pull/15376#issuecomment-1947428547
close as duplication
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16259 Immutable MetadataCache to improve client performance [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #15376:
URL: https://github.com/apache/kafka/pull/15376#issuecomment-1945065055
Thanks for the PR. Is this similar to what was done in #15323?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org