You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "dameiss-tibco (via GitHub)" <gi...@apache.org> on 2024/09/19 13:43:06 UTC
[I] [Bug] When using TLS to communicate to proxy/brokers, unable to delete a namespace after splitting bundles. Error does not occur when NOT using TLS. [pulsar]
dameiss-tibco opened a new issue, #23323:
URL: https://github.com/apache/pulsar/issues/23323
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
Linux c04-pulsar-toolset-0 6.10.0-linuxkit #1 SMP Wed Jul 17 10:51:09 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
Java 17.0.12
Pulsar 3.0.6
Running in Kubernetes cluster (host platform is MacOS ARM64) using Pulsar helm charts
### Minimal reproduce step
```
bin/pulsar-admin namespaces create public/namespace1
bin/pulsar-admin namespaces bundles public/namespace1
bin/pulsar-admin namespaces split-bundle --bundle 0x00000000_0x40000000 public/namespace1
bin/pulsar-admin namespaces bundles public/namespace1
bin/pulsar-admin namespaces split-bundle --bundle 0x40000000_0x80000000 public/namespace1
bin/pulsar-admin namespaces bundles public/namespace1
bin/pulsar-admin namespaces split-bundle --bundle 0x80000000_0xc0000000 --unload public/namespace1
bin/pulsar-admin namespaces bundles public/namespace1
bin/pulsar-admin namespaces delete public/namespace1
```
### What did you expect to see?
Succesful deletion of `public/namespace1`
### What did you see instead?
```
pulsar@c04-pulsar-toolset-0:/pulsar$ bin/pulsar-admin clusters get cluster-04
2024-09-19T13:28:06,782+0000 [main] WARN org.apache.pulsar.common.util.SecurityUtility - Conscrypt isn't available for Linux aarch64. Using JDK default security provider.
{
"serviceUrl" : "http://c04-pulsar-broker.msgmx.svc.cluster.local:8080/",
"serviceUrlTls" : "https://c04-pulsar-broker.msgmx.svc.cluster.local:8443/",
"brokerServiceUrl" : "pulsar://c04-pulsar-broker.msgmx.svc.cluster.local:/",
"brokerServiceUrlTls" : "pulsar+ssl://c04-pulsar-broker.msgmx.svc.cluster.local:6651/",
"brokerClientTlsEnabled" : false,
"tlsAllowInsecureConnection" : false,
"brokerClientTlsEnabledWithKeyStore" : false,
"brokerClientTlsTrustStoreType" : "JKS",
"brokerClientTlsKeyStoreType" : "JKS",
"migrated" : false
}
pulsar@c04-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces create public/namespace1
2024-09-19T13:23:55,951+0000 [main] WARN org.apache.pulsar.common.util.SecurityUtility - Conscrypt isn't available for Linux aarch64. Using JDK default security provider.
pulsar@c04-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces bundles public/namespace1
2024-09-19T13:24:05,660+0000 [main] WARN org.apache.pulsar.common.util.SecurityUtility - Conscrypt isn't available for Linux aarch64. Using JDK default security provider.
{
"boundaries" : [ "0x00000000", "0x40000000", "0x80000000", "0xc0000000", "0xffffffff" ],
"numBundles" : 4
}
pulsar@c04-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces split-bundle --bundle 0x00000000_0x40000000 public/namespace1
2024-09-19T13:24:14,505+0000 [main] WARN org.apache.pulsar.common.util.SecurityUtility - Conscrypt isn't available for Linux aarch64. Using JDK default security provider.
pulsar@c04-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces bundles public/namespace1
2024-09-19T13:24:24,323+0000 [main] WARN org.apache.pulsar.common.util.SecurityUtility - Conscrypt isn't available for Linux aarch64. Using JDK default security provider.
{
"boundaries" : [ "0x00000000", "0x20000000", "0x40000000", "0x80000000", "0xc0000000", "0xffffffff" ],
"numBundles" : 5
}
pulsar@c04-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces split-bundle --bundle 0x40000000_0x80000000 public/namespace1
2024-09-19T13:24:33,131+0000 [main] WARN org.apache.pulsar.common.util.SecurityUtility - Conscrypt isn't available for Linux aarch64. Using JDK default security provider.
pulsar@c04-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces bundles public/namespace1
2024-09-19T13:24:42,216+0000 [main] WARN org.apache.pulsar.common.util.SecurityUtility - Conscrypt isn't available for Linux aarch64. Using JDK default security provider.
{
"boundaries" : [ "0x00000000", "0x20000000", "0x40000000", "0x60000000", "0x80000000", "0xc0000000", "0xffffffff" ],
"numBundles" : 6
}
pulsar@c04-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces split-bundle --bundle 0x80000000_0xc0000000 --unload public/namespace1
2024-09-19T13:24:48,306+0000 [main] WARN org.apache.pulsar.common.util.SecurityUtility - Conscrypt isn't available for Linux aarch64. Using JDK default security provider.
pulsar@c04-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces bundles public/namespace1
2024-09-19T13:24:52,635+0000 [main] WARN org.apache.pulsar.common.util.SecurityUtility - Conscrypt isn't available for Linux aarch64. Using JDK default security provider.
{
"boundaries" : [ "0x00000000", "0x20000000", "0x40000000", "0x60000000", "0x80000000", "0xa0000000", "0xc0000000", "0xffffffff" ],
"numBundles" : 7
}
pulsar@c04-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces delete public/namespace1
2024-09-19T13:24:59,029+0000 [main] WARN org.apache.pulsar.common.util.SecurityUtility - Conscrypt isn't available for Linux aarch64. Using JDK default security provider.
2024-09-19T13:24:59,828+0000 [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [https://c04-pulsar-proxy:443/admin/v2/namespaces/public/namespace1?force=false] Failed to perform http delete request: javax.ws.rs.InternalServerErrorException: HTTP 500 {}
HTTP 500 {}
Reason: HTTP 500 {}
```
From the broker log:
```
2024-09-19T13:41:23,049+0000 [AsyncHttpClient-46-2] WARN org.apache.pulsar.client.admin.internal.BaseResource - [https://c04-pulsar-broker-0.c04-pulsar-broker.msgmx.svc.cluster.local:8443/admin/v2/persistent/public/namespace1/__change_events?force=true&deleteSchema=true] Failed to perform http delete request: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: General OpenSslEngine problem
2024-09-19T13:41:23,050+0000 [AsyncHttpClient-46-2] ERROR org.apache.pulsar.broker.admin.v2.Namespaces - [null] Failed to delete namespace public/namespace1
java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: General OpenSslEngine problem
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498) ~[?:?]
at java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
at org.apache.pulsar.client.admin.internal.BaseResource$4.failed(BaseResource.java:237) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.6.jar:3.0.6]
at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:247) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:242) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
at org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:62) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
at org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:227) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.6.jar:3.0.6]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:289) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.6.jar:3.0.6]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:181) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.channel.NettyConnectListener$1.onFailure(NettyConnectListener.java:151) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:26) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.111.Final.jar:4.1.111.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[io.netty-netty-common-4.1.111.Final.jar:4.1.111.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[io.netty-netty-common-4.1.111.Final.jar:4.1.111.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.111.Final.jar:4.1.111.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.111.Final.jar:4.1.111.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.111.Final.jar:4.1.111.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1359) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1341) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[io.netty-netty-codec-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[io.netty-netty-codec-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[io.netty-netty-codec-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty-netty-transport-4.1.111.Final.jar:4.1.111.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[io.netty-netty-common-4.1.111.Final.jar:4.1.111.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.111.Final.jar:4.1.111.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.111.Final.jar:4.1.111.Final]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: General OpenSslEngine problem
at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:300) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.6.jar:3.0.6]
... 56 more
Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: General OpenSslEngine problem
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
at java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1576) ~[?:?]
... 40 more
Caused by: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: General OpenSslEngine problem
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:291) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.6.jar:3.0.6]
... 37 more
Caused by: java.net.ConnectException: General OpenSslEngine problem
at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
... 31 more
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshakeException(ReferenceCountedOpenSslEngine.java:1927) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:848) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:564) ~[?:?]
at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1129) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:973) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1509) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
... 20 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
at io.netty.handler.ssl.EnhancingX509ExtendedTrustManager.checkServerTrusted(EnhancingX509ExtendedTrustManager.java:69) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:235) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:797) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.internal.tcnative.CertificateVerifierTask.runTask(CertificateVerifierTask.java:36) ~[io.netty-netty-tcnative-classes-2.0.65.Final.jar:2.0.65.Final]
at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:48) ~[io.netty-netty-tcnative-classes-2.0.65.Final.jar:2.0.65.Final]
at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:42) ~[io.netty-netty-tcnative-classes-2.0.65.Final.jar:2.0.65.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.runAndResetNeedTask(ReferenceCountedOpenSslEngine.java:1533) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.access$700(ReferenceCountedOpenSslEngine.java:94) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$TaskDecorator.run(ReferenceCountedOpenSslEngine.java:1505) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1649) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1495) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
... 20 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148) ~[?:?]
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129) ~[?:?]
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
at io.netty.handler.ssl.EnhancingX509ExtendedTrustManager.checkServerTrusted(EnhancingX509ExtendedTrustManager.java:69) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:235) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:797) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.internal.tcnative.CertificateVerifierTask.runTask(CertificateVerifierTask.java:36) ~[io.netty-netty-tcnative-classes-2.0.65.Final.jar:2.0.65.Final]
at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:48) ~[io.netty-netty-tcnative-classes-2.0.65.Final.jar:2.0.65.Final]
at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:42) ~[io.netty-netty-tcnative-classes-2.0.65.Final.jar:2.0.65.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.runAndResetNeedTask(ReferenceCountedOpenSslEngine.java:1533) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.access$700(ReferenceCountedOpenSslEngine.java:94) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$TaskDecorator.run(ReferenceCountedOpenSslEngine.java:1505) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1649) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1495) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336) ~[io.netty-netty-handler-4.1.111.Final.jar:4.1.111.Final]
... 20 more
2024-09-19T13:41:23,056+0000 [AsyncHttpClient-46-2] INFO org.eclipse.jetty.server.RequestLog - 10.1.4.18 - - [19/Sep/2024:13:41:22 +0000] "DELETE /admin/v2/namespaces/public/namespace1?force=false HTTP/1.1" 500 2 "-" "Pulsar-Java-v3.0.6" 135
```
### Anything else?
The same steps run on a non-TLS cluster:
```
pulsar@c03-pulsar-toolset-0:/pulsar$ bin/pulsar-admin clusters get cluster-03
{
"serviceUrl" : "http://c03-pulsar-broker.msgmx.svc.cluster.local:8080/",
"serviceUrlTls" : "https://c03-pulsar-broker.msgmx.svc.cluster.local:8443/",
"brokerServiceUrl" : "pulsar://c03-pulsar-broker.msgmx.svc.cluster.local:6650/",
"brokerServiceUrlTls" : "pulsar+ssl://c03-pulsar-broker.msgmx.svc.cluster.local:6651/",
"brokerClientTlsEnabled" : false,
"tlsAllowInsecureConnection" : false,
"brokerClientTlsEnabledWithKeyStore" : false,
"brokerClientTlsTrustStoreType" : "JKS",
"brokerClientTlsKeyStoreType" : "JKS",
"migrated" : false
}
pulsar@c03-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces create public/namespace1
pulsar@c03-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces bundles public/namespace1
{
"boundaries" : [ "0x00000000", "0x40000000", "0x80000000", "0xc0000000", "0xffffffff" ],
"numBundles" : 4
}
pulsar@c03-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces split-bundle --bundle 0x00000000_0x40000000 public/namespace1
pulsar@c03-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces bundles public/namespace1
{
"boundaries" : [ "0x00000000", "0x20000000", "0x40000000", "0x80000000", "0xc0000000", "0xffffffff" ],
"numBundles" : 5
}
pulsar@c03-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces split-bundle --bundle 0x40000000_0x80000000 public/namespace1
pulsar@c03-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces bundles public/namespace1
{
"boundaries" : [ "0x00000000", "0x20000000", "0x40000000", "0x60000000", "0x80000000", "0xc0000000", "0xffffffff" ],
"numBundles" : 6
}
pulsar@c03-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces split-bundle --bundle 0x80000000_0xc0000000 --unload public/namespace1
pulsar@c03-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces bundles public/namespace1
{
"boundaries" : [ "0x00000000", "0x20000000", "0x40000000", "0x60000000", "0x80000000", "0xa0000000", "0xc0000000", "0xffffffff" ],
"numBundles" : 7
}
pulsar@c03-pulsar-toolset-0:/pulsar$ bin/pulsar-admin namespaces delete public/namespace1
pulsar@c03-pulsar-toolset-0:/pulsar$
```
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When using TLS to communicate to proxy/brokers, unable to delete a namespace after splitting bundles. Error does not occur when NOT using TLS. [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #23323:
URL: https://github.com/apache/pulsar/issues/23323#issuecomment-6380981164
Thanks for the report, @dameiss-tibco. The broker log shows that the namespace deletion fails when the broker's internal admin client calls another broker over HTTPS (port 8443) and the TLS handshake fails with `PKIX path building failed: unable to find valid certification path to requested target`. That means the broker's client-side trust settings don't trust the other broker's certificate. Splitting with `--unload` most likely just moved a bundle to another broker, which made that broker-to-broker call necessary. Could you share the broker's TLS client settings (`brokerClientTlsEnabled`, `brokerClientTrustCertsFilePath` or `brokerClientTlsEnabledWithKeyStore` / `brokerClientTlsTrustStore`, `tlsTrustCertsFilePath`) and check that the CA configured there signed the brokers' certificates? 3.0.x is no longer supported, so please also try to reproduce on a currently supported release (4.0.x, 4.2.x or 5.0.x) and attach the broker logs if it still fails. If there's no update, this iss
ue will be closed in a few weeks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org