You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/08/11 07:28:02 UTC
[GitHub] [pulsar] lhotari opened a new issue #11637: [Broker] Failed to process metadata store notification
lhotari opened a new issue #11637:
URL: https://github.com/apache/pulsar/issues/11637
**Describe the bug**
I came a cross multiple odd log entries "Failed to process metadata store notification
java.lang.IllegalArgumentException: Invalid namespace format. expected <tenant>/<namespace> or <tenant>/<cluster>/<namespace> but got: public" when investigating #11635 .
I wonder if this has recently changed. This is the log entry:
```
06:28:50.626 [metadata-store-16-1] INFO org.apache.pulsar.common.naming.NamespaceBundleFactory - Policy updated for namespace public/dedup-2, refreshing the bundle cache.
06:28:50.626 [metadata-store-16-1] ERROR org.apache.pulsar.metadata.impl.AbstractMetadataStore - Failed to process metadata store notification
java.lang.IllegalArgumentException: Invalid namespace format. expected <tenant>/<namespace> or <tenant>/<cluster>/<namespace> but got: public
at org.apache.pulsar.common.naming.NamespaceName.<init>(NamespaceName.java:125) ~[org.apache.pulsar-pulsar-common-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
at org.apache.pulsar.common.naming.NamespaceName.<init>(NamespaceName.java:34) ~[org.apache.pulsar-pulsar-common-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
at org.apache.pulsar.common.naming.NamespaceName$1.load(NamespaceName.java:46) ~[org.apache.pulsar-pulsar-common-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
at org.apache.pulsar.common.naming.NamespaceName$1.load(NamespaceName.java:43) ~[org.apache.pulsar-pulsar-common-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:3951) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) ~[com.google.guava-guava-30.1-jre.jar:?]
at org.apache.pulsar.common.naming.NamespaceName.get(NamespaceName.java:69) ~[org.apache.pulsar-pulsar-common-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
at org.apache.pulsar.common.naming.NamespaceBundleFactory.handleMetadataStoreNotification(NamespaceBundleFactory.java:162) ~[org.apache.pulsar-pulsar-broker-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
at org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$0(AbstractMetadataStore.java:149) ~[org.apache.pulsar-pulsar-metadata-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:895) ~[?:1.8.0_292]
at org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$1(AbstractMetadataStore.java:147) ~[org.apache.pulsar-pulsar-metadata-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) [?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: java.lang.IllegalArgumentException: Invalid namespace format. namespace: public
at org.apache.pulsar.common.naming.NamespaceName.<init>(NamespaceName.java:122) ~[org.apache.pulsar-pulsar-common-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
... 20 more
```
**To Reproduce**
The above example is from the logs of running cpp-tests (see #11635). I assume that this happens in the master branch for all namespace policy updates.
**Expected behavior**
Namespace policy updates shouldn't result in errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11637: [Broker] Failed to process metadata store notification
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11637:
URL: https://github.com/apache/pulsar/issues/11637#issuecomment-899296289
Closing since this is fixed by #11656
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #11637: [Broker] Failed to process metadata store notification
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #11637:
URL: https://github.com/apache/pulsar/issues/11637#issuecomment-899287966
> This problem seems to be fixed by #11656 . @Technoboy- can you confirm?
Yes, I think so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari closed issue #11637: [Broker] Failed to process metadata store notification
Posted by GitBox <gi...@apache.org>.
lhotari closed issue #11637:
URL: https://github.com/apache/pulsar/issues/11637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11637: [Broker] Failed to process metadata store notification
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11637:
URL: https://github.com/apache/pulsar/issues/11637#issuecomment-899016978
This problem seems to be fixed by #11656 . @Technoboy- can you confirm?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11637: [Broker] Failed to process metadata store notification
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11637:
URL: https://github.com/apache/pulsar/issues/11637#issuecomment-896573110
this seems to always fail:
https://github.com/apache/pulsar/blob/794aa20d9f2a4e668cc36465362d22e042e6e536/pulsar-broker/src/main/java/org/apache/pulsar/common/naming/NamespaceBundleFactory.java#L160-L172
this logic looks risky:
https://github.com/apache/pulsar/blob/794aa20d9f2a4e668cc36465362d22e042e6e536/pulsar-broker/src/main/java/org/apache/pulsar/common/naming/NamespaceBundleFactory.java#L330-L343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11637: [Broker] Failed to process metadata store notification
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11637:
URL: https://github.com/apache/pulsar/issues/11637#issuecomment-896571001
@merlimat This problem might be related to recent PIP-45 changes. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #11637: [Broker] Failed to process metadata store notification
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #11637:
URL: https://github.com/apache/pulsar/issues/11637#issuecomment-896581747
I see.
let's fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #11637: [Broker] Failed to process metadata store notification
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #11637:
URL: https://github.com/apache/pulsar/issues/11637#issuecomment-896577249
I am pretty sure that CI passed when I merged the #11198 patch
probably there is some case that happens rarely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11637: [Broker] Failed to process metadata store notification
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11637:
URL: https://github.com/apache/pulsar/issues/11637#issuecomment-896579533
> I am pretty sure that CI passed when I merged the #11198 patch
>
> probably there is some case that happens rarely
It looks like a problem that just gets logged. The problem seems to be that `NotificationType.ChildrenChanged` isn't handled in [`NamespaceBundleFactory.handleMetadataStoreNotification`](https://github.com/apache/pulsar/blob/794aa20d9f2a4e668cc36465362d22e042e6e536/pulsar-broker/src/main/java/org/apache/pulsar/common/naming/NamespaceBundleFactory.java#L160-L172).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org