You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/09/29 18:08:00 UTC
[GitHub] [pulsar] klwilson227 opened a new issue #8163: Broker
klwilson227 opened a new issue #8163:
URL: https://github.com/apache/pulsar/issues/8163
**Describe the bug**
When upgrading a 2.5.2 instance of broker contained in a Kuberenetes Deployment to 2.6.1 with the broker contained in a deployment. The brokers do not function after the upgrade. One of the brokers, we see a error in the logs showing that it is attempting to use the hostname of another broker. Deployments in kuberenetes 1.19 do not support DNS resolution of the pods for inner pod communications this should be done with IP addresses.
**To Reproduce**
Steps to reproduce the behavior:
1. Using the public helm charts for pulsar, make the following modifications:
mv broker-statefulset.yaml to broker-deployment.yaml
Make the following changes to the yaml:
1. Remove spec.serviceName, spec.updateStrategy, spec.podManagementPolicy
2. Change kind: Deployment
Deploy all of the charts.
2. Attempt to push data to the proxy.
3. Open the logs for each of the brokers and scan for errors.
**Expected behavior**
Expect that the broker runs as a stateless deployment without errors.
**Screenshots**
16:57:39.970 [AsyncHttpClient-53-1] ERROR org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [itom-di-message-bus-svc] Failed to get list of topics under namespace public/default
java.util.concurrent.ExecutionException: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: itomdipulsar-broker-55fbf44874-lbbpm
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_252]
at org.apache.pulsar.broker.admin.v2.NonPersistentTopics.lambda$getList$1(NonPersistentTopics.java:275) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1]
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_252]
at org.apache.pulsar.client.admin.internal.TopicsImpl$5.failed(TopicsImpl.java:226) ~[org.apache.pulsar-pulsar-client-admin-original-2.6.1.jar:2.6.1]
at org.glassfish.jersey.client.JerseyInvocation$4.failed(JerseyInvocation.java:1030) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:231) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
at org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:85) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
at org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:183) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
at org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:183) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:200) ~[org.apache.pulsar-pulsar-client-admin-original-2.6.1.jar:2.6.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_252]
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$3(AsyncHttpConnector.java:242) ~[org.apache.pulsar-pulsar-client-admin-original-2.6.1.jar:2.6.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) [?:1.8.0_252]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) [?:1.8.0_252]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) [?:1.8.0_252]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) [?:1.8.0_252]
at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:473) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.request.NettyRequestSender$1.onFailure(NettyRequestSender.java:325) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:26) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:183) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:301) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithCertainForceConnect(NettyRequestSender.java:142) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:113) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.request.NettyRequestSender.sendNextRequest(NettyRequestSender.java:548) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.handler.intercept.Redirect30xInterceptor.exitAfterHandlingRedirect(Redirect30xInterceptor.java:160) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.handler.intercept.Interceptors.exitAfterIntercept(Interceptors.java:98) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.handler.HttpHandler.handleHttpResponse(HttpHandler.java:78) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:140) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) [io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) [io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) [io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) [io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282) [io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) [io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) [io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: itomdipulsar-broker-55fbf44874-lbbpm
at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:233) ~[org.apache.pulsar-pulsar-client-admin-original-2.6.1.jar:2.6.1]
... 76 more
Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: itomdipulsar-broker-55fbf44874-lbbpm
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_252]
**Desktop (please complete the following information):**
- kubernetes 1.19
**Additional context**
Note the hostname itomdipulsar-broker-55fbf44874-lbbpm is used in the communications vs. the IP address.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #8163: Broker
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #8163:
URL: https://github.com/apache/pulsar/issues/8163#issuecomment-701117706
@klwilson227 It sounds like a DNS resolution issue, not a Pulsar issue.
You can configure `advertisedAddress` to podIP in brokers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] klwilson227 closed issue #8163: Broker
Posted by GitBox <gi...@apache.org>.
klwilson227 closed issue #8163:
URL: https://github.com/apache/pulsar/issues/8163
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org