You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "BewareMyPower (via GitHub)" <gi...@apache.org> on 2026/02/13 14:48:29 UTC
[PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
BewareMyPower opened a new pull request, #25246:
URL: https://github.com/apache/pulsar/pull/25246
### Motivation
https://github.com/apache/pulsar/pull/25187 introduces a regression on `MetadataCacheImpl#refresh`. When a path is created or modified, the `refresh` method will be called by updating `objCache` with the future returned by `readValueFromStore()`. However, there is a race:
1. [current thread] Acquire the lock of the node in the `ConcurrentMap` of `objCache`. Then call `readValueFromStore`, which calls `store.get`
2. [metadata store worker thread] In the callback of `store.get`, `objCache.getIfPresent` calls the `ConcurrentMap#get` to get the cached future and check if it's the same with the future inserted to the map
3. [current thread] Insert the future returned by to the `ConcurrentMap` of `objCache` and release the lock
The updated future of step 3 is not guaranteed to be immediately visible because `ConcurrentMap#get` is lock-free, which means it does not need to wait the lock on `path` is released after step 3.
https://github.com/apache/pulsar/blob/9db31cca40e3948182386b6930907daf449abd41/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/cache/impl/MetadataCacheImpl.java#L156-L162
The correctness of the code above is based on the fact that the `cachedFuture` must be the future of `readValueFromStore`, but not the existing cached future.
This logic was added originally because `testCloneInReadModifyUpdateOrCreate` failed. I've thought it's caused by duplicated `refresh` calls but it's actually not. However, the root cause is that when it failed, the `create` method implementation was wrong:
```java
return serialize(path, value).thenAcceptAsync(content -> store.put(path, content, Optional.of(-1L)))
.thenApply(stat -> objCache.get(path))
/* ... */
.thenApply(__ -> null);
```
It should be `thenComposeAsync` rather than `thenAcceptAsync`, otherwise `objCache.get` might not see the updated value. This bug was accidentally fixed in my later commits.
### Modifications
- Revert the `cachedFuture` related code. We might need a careful design to reduce unnecessary deserializations when `refresh` is called multiple times for an update (e.g. the callback of `MetadataCacheImpl#put` and the notification method of `AbstractMetadataStore#accept`)
- Add `testRefreshRace` to verify this race disappears.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #25246:
URL: https://github.com/apache/pulsar/pull/25246#discussion_r2811977750
##########
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/MetadataCacheTest.java:
##########
@@ -739,4 +741,18 @@ public void testNoBackoffMetadataCacheConfig() {
assertTrue(backoff.isMandatoryStopMade());
assertEquals(backoff.getFirstBackoffTimeInMillis(), 0);
}
+
+ @Test
+ public void testRefreshRace() throws Exception {
+ @Cleanup final var store = new LocalMemoryMetadataStore("memory:local", MetadataStoreConfig.builder().build());
+ final var cache = store.getMetadataCache(String.class);
+ for (int i = 0; i < 500; i++) {
+ final var key = "/key" + i;
+ assertTrue(cache.get(key).get().isEmpty());
+
+ store.put(key, "\"value\"".getBytes(StandardCharsets.UTF_8), Optional.empty()).get();
+ Awaitility.await().pollInterval(Duration.ofMillis(1)).atMost(Duration.ofSeconds(3)).untilAsserted(() ->
Review Comment:
I intentionally called `store.put` rather than `cache.put`. This test is added mainly to verify even if the side effect only comes from the notification listener of `AbstractMetadataStore#accept`, it still works. Hence it should not rely on the semantics of the MetadataCache `put` API, which refreshes the cache before the future is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25246:
URL: https://github.com/apache/pulsar/pull/25246#issuecomment-3907502006
## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/25246?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 72.59%. Comparing base ([`9db31cc`](https://app.codecov.io/gh/apache/pulsar/commit/9db31cca40e3948182386b6930907daf449abd41?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b9b627c`](https://app.codecov.io/gh/apache/pulsar/commit/b9b627c5d1f3abc616fff0d5cf67d05c237eb4ef?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
[](https://app.codecov.io/gh/apache/pulsar/pull/25246?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
```diff
@@ Coverage Diff @@
## master #25246 +/- ##
=============================================
+ Coverage 37.96% 72.59% +34.63%
- Complexity 13510 34396 +20886
=============================================
Files 1902 1959 +57
Lines 151242 155401 +4159
Branches 17240 17724 +484
=============================================
+ Hits 57413 112821 +55408
+ Misses 86193 33576 -52617
- Partials 7636 9004 +1368
```
| [Flag](https://app.codecov.io/gh/apache/pulsar/pull/25246/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [inttests](https://app.codecov.io/gh/apache/pulsar/pull/25246/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `25.61% <ø> (-0.49%)` | :arrow_down: |
| [systests](https://app.codecov.io/gh/apache/pulsar/pull/25246/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `22.38% <ø> (-0.20%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pulsar/pull/25246/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.57% <ø> (+38.96%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/25246?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [.../pulsar/metadata/cache/impl/MetadataCacheImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/25246?src=pr&el=tree&filepath=pulsar-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fmetadata%2Fcache%2Fimpl%2FMetadataCacheImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLW1ldGFkYXRhL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvbWV0YWRhdGEvY2FjaGUvaW1wbC9NZXRhZGF0YUNhY2hlSW1wbC5qYXZh) | `85.29% <ø> (+7.57%)` | :arrow_up: |
... and [1410 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/25246/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #25246:
URL: https://github.com/apache/pulsar/pull/25246#discussion_r2810915058
##########
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/MetadataCacheTest.java:
##########
@@ -739,4 +741,18 @@ public void testNoBackoffMetadataCacheConfig() {
assertTrue(backoff.isMandatoryStopMade());
assertEquals(backoff.getFirstBackoffTimeInMillis(), 0);
}
+
+ @Test
+ public void testRefreshRace() throws Exception {
+ @Cleanup final var store = new LocalMemoryMetadataStore("memory:local", MetadataStoreConfig.builder().build());
+ final var cache = store.getMetadataCache(String.class);
+ for (int i = 0; i < 500; i++) {
+ final var key = "/key" + i;
+ assertTrue(cache.get(key).get().isEmpty());
+
+ store.put(key, "\"value\"".getBytes(StandardCharsets.UTF_8), Optional.empty()).get();
+ Awaitility.await().pollInterval(Duration.ofMillis(1)).atMost(Duration.ofSeconds(3)).untilAsserted(() ->
Review Comment:
The `put` operation is called on the `MetadataStore` object. The `MetadataCache#accept` method is not guaranteed to be called immediately after the `put` operation is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #25246:
URL: https://github.com/apache/pulsar/pull/25246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #25246:
URL: https://github.com/apache/pulsar/pull/25246#discussion_r2810915058
##########
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/MetadataCacheTest.java:
##########
@@ -739,4 +741,18 @@ public void testNoBackoffMetadataCacheConfig() {
assertTrue(backoff.isMandatoryStopMade());
assertEquals(backoff.getFirstBackoffTimeInMillis(), 0);
}
+
+ @Test
+ public void testRefreshRace() throws Exception {
+ @Cleanup final var store = new LocalMemoryMetadataStore("memory:local", MetadataStoreConfig.builder().build());
+ final var cache = store.getMetadataCache(String.class);
+ for (int i = 0; i < 500; i++) {
+ final var key = "/key" + i;
+ assertTrue(cache.get(key).get().isEmpty());
+
+ store.put(key, "\"value\"".getBytes(StandardCharsets.UTF_8), Optional.empty()).get();
+ Awaitility.await().pollInterval(Duration.ofMillis(1)).atMost(Duration.ofSeconds(3)).untilAsserted(() ->
Review Comment:
The `put` operation is called on the `MetadataStore` object. The `MetadataCache#accept` method is not guaranteed to be called immediately after the `put` operation is done because the notification callback is executed in the dedicated `metadata-store-event` thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #25246:
URL: https://github.com/apache/pulsar/pull/25246#issuecomment-3897664538
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #25246:
URL: https://github.com/apache/pulsar/pull/25246#issuecomment-3900519822
@lhotari Do you know why `ResourceQuotaCalculatorImplTest` now run on Broker Group 2? I checked the previous workflows and found it should run in BROKER_FLAKY workflow. https://github.com/apache/pulsar/actions/runs/21906841628/job/63249299032
Anyway, I pushed a PR to fix this broken test (not flaky): https://github.com/apache/pulsar/pull/25247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #25246:
URL: https://github.com/apache/pulsar/pull/25246#discussion_r2810915058
##########
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/MetadataCacheTest.java:
##########
@@ -739,4 +741,18 @@ public void testNoBackoffMetadataCacheConfig() {
assertTrue(backoff.isMandatoryStopMade());
assertEquals(backoff.getFirstBackoffTimeInMillis(), 0);
}
+
+ @Test
+ public void testRefreshRace() throws Exception {
+ @Cleanup final var store = new LocalMemoryMetadataStore("memory:local", MetadataStoreConfig.builder().build());
+ final var cache = store.getMetadataCache(String.class);
+ for (int i = 0; i < 500; i++) {
+ final var key = "/key" + i;
+ assertTrue(cache.get(key).get().isEmpty());
+
+ store.put(key, "\"value\"".getBytes(StandardCharsets.UTF_8), Optional.empty()).get();
+ Awaitility.await().pollInterval(Duration.ofMillis(1)).atMost(Duration.ofSeconds(3)).untilAsserted(() ->
Review Comment:
The `put` operation is called on the `MetadataStore` object. The `MetadataCache#accept` method is not guaranteed to be called immediately after the `put` operation is done because the notification callback is executed in the dedicated `metadata-store-event` thread.
See PIP-453 for the new thread model (though before PIP-453, it's also executed in that thread)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
Posted by "gaoran10 (via GitHub)" <gi...@apache.org>.
gaoran10 commented on code in PR #25246:
URL: https://github.com/apache/pulsar/pull/25246#discussion_r2807701715
##########
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/MetadataCacheTest.java:
##########
@@ -739,4 +741,18 @@ public void testNoBackoffMetadataCacheConfig() {
assertTrue(backoff.isMandatoryStopMade());
assertEquals(backoff.getFirstBackoffTimeInMillis(), 0);
}
+
+ @Test
+ public void testRefreshRace() throws Exception {
+ @Cleanup final var store = new LocalMemoryMetadataStore("memory:local", MetadataStoreConfig.builder().build());
+ final var cache = store.getMetadataCache(String.class);
+ for (int i = 0; i < 500; i++) {
+ final var key = "/key" + i;
+ assertTrue(cache.get(key).get().isEmpty());
+
+ store.put(key, "\"value\"".getBytes(StandardCharsets.UTF_8), Optional.empty()).get();
+ Awaitility.await().pollInterval(Duration.ofMillis(1)).atMost(Duration.ofSeconds(3)).untilAsserted(() ->
Review Comment:
Why does the test need to wait here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #25246:
URL: https://github.com/apache/pulsar/pull/25246#discussion_r2811601984
##########
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/MetadataCacheTest.java:
##########
@@ -739,4 +741,18 @@ public void testNoBackoffMetadataCacheConfig() {
assertTrue(backoff.isMandatoryStopMade());
assertEquals(backoff.getFirstBackoffTimeInMillis(), 0);
}
+
+ @Test
+ public void testRefreshRace() throws Exception {
+ @Cleanup final var store = new LocalMemoryMetadataStore("memory:local", MetadataStoreConfig.builder().build());
+ final var cache = store.getMetadataCache(String.class);
+ for (int i = 0; i < 500; i++) {
+ final var key = "/key" + i;
+ assertTrue(cache.get(key).get().isEmpty());
+
+ store.put(key, "\"value\"".getBytes(StandardCharsets.UTF_8), Optional.empty()).get();
+ Awaitility.await().pollInterval(Duration.ofMillis(1)).atMost(Duration.ofSeconds(3)).untilAsserted(() ->
Review Comment:
btw. Wait could be avoided when `cache.put` is used. Internally, it will wait until the update has completed and then refresh the cache by reading from the store again. However I don't think it's necessary to add this test in this PR since I'd like to get 4.0.9 release candidate 2 and 4.1.3 release candidate 2 started asap.
```java
@Test
public void testRefreshRaceCacheOnly() throws Exception {
@Cleanup final var store = new LocalMemoryMetadataStore("memory:local", MetadataStoreConfig.builder().build());
final var cache = store.getMetadataCache(String.class);
for (int i = 0; i < 500; i++) {
final var key = "/key" + i;
assertTrue(cache.get(key).get().isEmpty());
cache.put(key, "\"value\"", EnumSet.noneOf(CreateOption.class)).get();
assertTrue(cache.get(key).get().isPresent(), "Failed at key " + key);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][meta] Metadata cache refresh might not take effect [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #25246:
URL: https://github.com/apache/pulsar/pull/25246#discussion_r2811977750
##########
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/MetadataCacheTest.java:
##########
@@ -739,4 +741,18 @@ public void testNoBackoffMetadataCacheConfig() {
assertTrue(backoff.isMandatoryStopMade());
assertEquals(backoff.getFirstBackoffTimeInMillis(), 0);
}
+
+ @Test
+ public void testRefreshRace() throws Exception {
+ @Cleanup final var store = new LocalMemoryMetadataStore("memory:local", MetadataStoreConfig.builder().build());
+ final var cache = store.getMetadataCache(String.class);
+ for (int i = 0; i < 500; i++) {
+ final var key = "/key" + i;
+ assertTrue(cache.get(key).get().isEmpty());
+
+ store.put(key, "\"value\"".getBytes(StandardCharsets.UTF_8), Optional.empty()).get();
+ Awaitility.await().pollInterval(Duration.ofMillis(1)).atMost(Duration.ofSeconds(3)).untilAsserted(() ->
Review Comment:
I intentionally called `store.put` rather than `cache.put`. This test is added mainly to verify the notification listener from `AbstractMetadataStore#accept` works, so it should not rely on the semantics of the MetadataCache `put` API, which refreshes the cache before the future is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org