You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/10/15 05:26:58 UTC

[GitHub] [pulsar] k2la opened a new issue #5387: When I create many PulsarClients and Consumers, failed to resolve hostname

k2la opened a new issue #5387: When I create many PulsarClients and Consumers, failed to resolve hostname
URL: https://github.com/apache/pulsar/issues/5387
 
 
   I have use cases that needs to create many  `PulsarClient` and `Consumer` instances.
   
   ```
   List<PulsarClient> clients = new ArrayList<>();
   List<Consumer<byte[]>> consumers = new ArrayList<>();
     for (int i = 0; i < 100; i++) {
       clients.add(PulsarClient.builder()
         .serviceUrl(url)
         .build());
       consumers.add(clients.get(i).newConsumer()
         .topic(topic)
         .subscriptionName(subs)
         .messageListener(new MessageListener<byte[]>() {
           @Override
           public void received(Consumer<byte[]> consumer, Message<byte[]> message) {
             System.out.println("Received: " + new String(message.getData()));
             consumer.acknowledgeAsync(message);
         	}
         })
         .subscriptionType(SubscriptionType.Shared)
         .subscribe());
     }
   ```
   
   When I set MaxDirectMemorySize of JVM 1GB and create 100 each `PulsarClient` and `Consumer` instances like above code, the client program failed to resolve hostname with the following logs.
   
    ```
   [pulsar-client-io-164-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to example.broker.dev.pulsar.jp:6650 : org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'example.broker.dev.pulsar.jp' failed to resolve 'example.broker.dev.pulsar.jp' after 4 queries
   [pulsar-client-io-164-1] WARN org.apache.pulsar.client.impl.PulsarClientImpl - [persistent://tenant1/ns1/t1] Failed to get partitioned topic metadata
   java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'example.broker.dev.pulsar.jp' failed to resolve 'example.broker.dev.pulsar.jp' after 4 queries
   	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
   	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
   	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:647)
   	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
   	at org.apache.pulsar.client.impl.ConnectionPool.lambda$null$9(ConnectionPool.java:202)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:335)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   	at java.lang.Thread.run(Thread.java:748)
   Caused by: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'example.broker.dev.pulsar.jp' failed to resolve 'example.broker.dev.pulsar.jp' after 4 queries
   	... 7 more
   Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'example.broker.dev.pulsar.jp' failed to resolve 'example.broker.dev.pulsar.jp' after 4 queries
   	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
   	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
   	at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943)
   	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
   	at org.apache.pulsar.client.impl.ConnectionPool.lambda$resolveName$16(ConnectionPool.java:259)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:198)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:856)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:809)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:332)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.access$600(DnsResolveContext.java:62)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$3.operationComplete(DnsResolveContext.java:381)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:162)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:33)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:356)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.access$600(DnsResolveContext.java:62)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$3.operationComplete(DnsResolveContext.java:381)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext.setFailure(DnsQueryContext.java:220)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext.onQueryWriteCompletion(DnsQueryContext.java:155)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext.access$200(DnsQueryContext.java:43)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:147)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:144)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121)
   	at org.apache.pulsar.shade.io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64)
   	at org.apache.pulsar.shade.io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:709)
   	at org.apache.pulsar.shade.io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:294)
   	at org.apache.pulsar.shade.io.netty.channel.ChannelOutboundBuffer.failFlushed(ChannelOutboundBuffer.java:646)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.closeOutboundBufferForShutdown(AbstractChannel.java:685)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.shutdownOutput(AbstractChannel.java:678)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:952)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:512)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:905)
   	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1396)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
   	at org.apache.pulsar.shade.io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
   	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:1013)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel.flush(AbstractChannel.java:252)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver.flushQueries(DnsNameResolver.java:1094)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.internalResolve(DnsResolveContext.java:276)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.doSearchDomainQuery(DnsResolveContext.java:244)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsAddressResolveContext.doSearchDomainQuery(DnsAddressResolveContext.java:98)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.resolve(DnsResolveContext.java:211)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver.doResolveAllUncached0(DnsNameResolver.java:995)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver.access$500(DnsNameResolver.java:84)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver$5.run(DnsNameResolver.java:979)
   	... 6 more
   Caused by: org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'example.broker.dev.pulsar.jp' failed to resolve 'example.broker.dev.pulsar.jp' after 4 queries
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:848)
   	... 59 more
   Caused by: org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolverException: [/172.30.2.9:53] failed to send a query (no stack trace available)
   Caused by: java.nio.channels.ClosedChannelException
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
   Exception in thread "main" org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'example.broker.dev.pulsar.jp' failed to resolve 'example.broker.dev.pulsar.jp' after 4 queries
   	at org.apache.pulsar.client.impl.ConnectionPool.lambda$null$9(ConnectionPool.java:202)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:335)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   	at java.lang.Thread.run(Thread.java:748)
   Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'example.broker.dev.pulsar.jp' failed to resolve 'example.broker.dev.pulsar.jp' after 4 queries
   	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
   	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
   	at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943)
   	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
   	at org.apache.pulsar.client.impl.ConnectionPool.lambda$resolveName$16(ConnectionPool.java:259)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:198)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:856)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:809)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:332)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.access$600(DnsResolveContext.java:62)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$3.operationComplete(DnsResolveContext.java:381)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:162)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:33)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:356)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.access$600(DnsResolveContext.java:62)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$3.operationComplete(DnsResolveContext.java:381)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext.setFailure(DnsQueryContext.java:220)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext.onQueryWriteCompletion(DnsQueryContext.java:155)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext.access$200(DnsQueryContext.java:43)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:147)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:144)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121)
   	at org.apache.pulsar.shade.io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64)
   	at org.apache.pulsar.shade.io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:709)
   	at org.apache.pulsar.shade.io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:294)
   	at org.apache.pulsar.shade.io.netty.channel.ChannelOutboundBuffer.failFlushed(ChannelOutboundBuffer.java:646)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.closeOutboundBufferForShutdown(AbstractChannel.java:685)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.shutdownOutput(AbstractChannel.java:678)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:952)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:512)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:905)
   	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1396)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
   	at org.apache.pulsar.shade.io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
   	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:1013)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel.flush(AbstractChannel.java:252)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver.flushQueries(DnsNameResolver.java:1094)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.internalResolve(DnsResolveContext.java:276)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.doSearchDomainQuery(DnsResolveContext.java:244)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsAddressResolveContext.doSearchDomainQuery(DnsAddressResolveContext.java:98)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.resolve(DnsResolveContext.java:211)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver.doResolveAllUncached0(DnsNameResolver.java:995)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver.access$500(DnsNameResolver.java:84)
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver$5.run(DnsNameResolver.java:979)
   	... 6 more
   Caused by: org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'example.broker.dev.pulsar.jp' failed to resolve 'example.broker.dev.pulsar.jp' after 4 queries
   	at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:848)
   	... 59 more
   Caused by: org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolverException: [/172.30.2.9:53] failed to send a query (no stack trace available)
   Caused by: java.nio.channels.ClosedChannelException
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
    ```
   
   How do I resolve this ?
   
   ### Steps to reproduce
   - Set MaxDirectMemorySize of JVM 1GB and create 100 each `PulsarClient` and `Consumer` instances like above code
   
   A client program didn't print above logs when connected standalone.
   
   #### System configuration
   Pulsar version of server and client library: 2.3.2
   Java version: 1.8
   OS: Linux

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services