You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/07/02 17:44:41 UTC
[GitHub] [pulsar] bharanic-dev opened a new pull request #11202: [Issue 11200] [pulsar-metadata] Fix for Metadata cache inconsistency across brokers
bharanic-dev opened a new pull request #11202:
URL: https://github.com/apache/pulsar/pull/11202
Fixes #11200
### Motivation
When a metadata resource is updated via pulsar-admin command, on brokers that did not handle the admin API, the cache can get in to an inconsistent state. getData() will then result in a stale cache data to be returned. And the broker never gets the update later on.
### Modifications
In the zookeeper watch callback, there is a race condition between the cache getting updated via the refresh() call here.
https://github.com/apache/pulsar/blob/c080c845d7b3a680e7af9511ef38aff3781185a8/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/cache/impl/MetadataCacheImpl.java#L271
If a subsystem does a getDataAsync() in the watch callback, the data returned could be stale data in the cache. The bigger issue is that this broker never gets a chance to see the updated data (since the watch event is already processed).
Fix is to invalidate the cache before triggering the refresh. This forces the getData()/getDataAsync() to fetch the latest data from zookeeper.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
- [x] verified manually the steps described in #11200 works after the change.
### Documentation
- Does this pull request introduce a new feature? (no)
- this is a bug fix, there is no documentation impact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jerrypeng merged pull request #11202: [Issue 11200] [pulsar-metadata] Unit test to verify Metadata cache consistency across brokers
Posted by GitBox <gi...@apache.org>.
jerrypeng merged pull request #11202:
URL: https://github.com/apache/pulsar/pull/11202
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jerrypeng commented on a change in pull request #11202: [Issue 11200] [pulsar-metadata] Fix for Metadata cache inconsistency across brokers
Posted by GitBox <gi...@apache.org>.
jerrypeng commented on a change in pull request #11202:
URL: https://github.com/apache/pulsar/pull/11202#discussion_r663194054
##########
File path: pulsar-metadata/src/main/java/org/apache/pulsar/metadata/cache/impl/MetadataCacheImpl.java
##########
@@ -268,6 +268,7 @@ public void accept(Notification t) {
case Modified:
if (objCache.synchronous().getIfPresent(path) != null) {
// Trigger background refresh of the cached item
+ objCache.synchronous().invalidate(path);
Review comment:
can you please a comment to why this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] bharanic-dev commented on pull request #11202: [Issue 11200] [pulsar-metadata] Fix for Metadata cache inconsistency across brokers
Posted by GitBox <gi...@apache.org>.
bharanic-dev commented on pull request #11202:
URL: https://github.com/apache/pulsar/pull/11202#issuecomment-873160024
@merlimat @jerrypeng please help review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] bharanic-dev commented on a change in pull request #11202: [Issue 11200] [pulsar-metadata] Unit test to verify Metadata cache consistency across brokers
Posted by GitBox <gi...@apache.org>.
bharanic-dev commented on a change in pull request #11202:
URL: https://github.com/apache/pulsar/pull/11202#discussion_r676987760
##########
File path: pulsar-metadata/src/main/java/org/apache/pulsar/metadata/cache/impl/MetadataCacheImpl.java
##########
@@ -268,6 +268,7 @@ public void accept(Notification t) {
case Modified:
if (objCache.synchronous().getIfPresent(path) != null) {
// Trigger background refresh of the cached item
+ objCache.synchronous().invalidate(path);
Review comment:
this change was subsumed by changes committed in #11267. This PR now only has the unit test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org