You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/06/12 05:00:37 UTC
[GitHub] [druid] zenfenan edited a comment on issue #325: Broker can lock up due to defunct http connections
zenfenan edited a comment on issue #325:
URL: https://github.com/apache/druid/issues/325#issuecomment-643065616
I think it still exists in 0.18.1. We have three brokers. After we run benchmark with some heavy queries, I see one of the broker getting into a stale/unresponsive state, meaning, the broker process is running on that node but it doesn't respond. I tried executing query directly on this broker which times out and even /status check doesn't produce a response. When I stop the broker process to restart the service again, I see a bunch of `WARN` log messages about jetty `QueuedThreadPool` and an `ERROR` saying `JettyServerModule - Jetty lifecycle event failed`.
The complete error log is given below.
```
2020-06-11T08:26:27,889 INFO [Thread-43] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/BROKER/prod-euclid-druid-queryserver1-1735672:8082]
2020-06-11T08:26:27,899 INFO [NodeRoleWatcher[BROKER]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Node[http://prod-euclid-druid-queryserver1-1735672:8082] of role[broker] went offline.
2020-06-11T08:26:27,900 INFO [Thread-43] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{"druidNode":{"service":"druid/broker","host":"prod-euclid-druid-queryserver1-1735672","bindOnHost":false,"plaintextPort":8082,"port":-1,"tlsPort":-1,"enablePlaintextPort":true,"enableTlsPort":false},"nodeType":"broker","services":{"lookupNodeService":{"type":"lookupNodeService","lookupTier":"__default"}}}].
2020-06-11T08:26:27,901 INFO [Thread-43] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2020-06-11T08:26:57,908 INFO [Thread-43] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@3909a854{HTTP/1.1,[http/1.1]}{0.0.0.0:8082}
2020-06-11T08:26:57,909 INFO [Thread-43] org.eclipse.jetty.server.session - node0 Stopped scavenging
2020-06-11T08:26:57,913 INFO [Thread-43] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@186dcb05{/,null,UNAVAILABLE}
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-162[timeseries_[euclid_cdo_live_hourly]_18f82248-c0a4-4a1b-b163-7a9626c9d91b],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-165[topN_[euclid_cdo_live_hourly]_549ddbd3-f42e-4f69-b75f-ee053d5bb0d5],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-177[topN_[euclid_cdo_live_hourly]_c99ddc98-0077-4d09-a53d-efd9f21eb5d9],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-159[topN_[euclid_cdo_live_hourly]_2cbbaac8-171b-4039-b8ec-78439c0ca563],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-175[topN_[euclid_cdo_live_hourly]_748433e0-294f-4df3-837b-3836b20d5ef5],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-163[topN_[euclid_cdo_live_hourly]_e41aeaac-5830-4e3e-9f78-1c5fbfca4656],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-182[topN_[euclid_cdo_live_hourly]_0381d5c9-d858-4c9a-aa57-6353e0334453],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-150[topN_[euclid_cdo_live_hourly]_51dd4cda-9fcf-4533-af6f-71d11568e4ae],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-169[topN_[euclid_cdo_live_hourly]_9a40b9f3-e77c-4393-af88-a949df285d61],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-181[topN_[euclid_cdo_live_hourly]_f3a2af6d-78cd-4ff5-8df6-909edb8040bb],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-184[topN_[euclid_cdo_live_hourly]_4ef89502-6e54-436a-b54e-47776a11ee31],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-155[topN_[euclid_cdo_live_hourly]_a4e2f32b-62a7-40b1-9160-54b36205908d],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-173[topN_[euclid_cdo_live_hourly]_c38ae97c-fb65-4b12-91c2-3fb8511d3939],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-180[topN_[euclid_cdo_live_hourly]_2049db4c-6796-4a69-acbc-f38a4fa1a666],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-149[topN_[euclid_cdo_live_hourly]_85f8f14e-7eda-4176-9408-b723728f2a8e],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-179[topN_[euclid_cdo_live_hourly]_6d83af84-35eb-4ecd-b40a-4e615061e0a0],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-174[topN_[euclid_cdo_live_hourly]_1e84dba4-80d5-4bbf-bc52-7373a3adc2fe],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-186[topN_[euclid_cdo_live_hourly]_2c476c10-7e41-45f7-8437-02bd6907d96d],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-161[topN_[euclid_cdo_live_hourly]_312791d2-704d-4b76-8104-8a87c2baa42a],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-166[topN_[euclid_cdo_live_hourly]_a2a2ddc7-cd92-4f99-85af-77cb4b7d050c],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-168[topN_[euclid_cdo_live_hourly]_b352dcb8-6dbc-4b1c-83bf-54f37427a122],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-171[topN_[euclid_cdo_live_hourly]_65d58a84-b67d-444a-b261-5947dec1def9],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-178[topN_[euclid_cdo_live_hourly]_37b5c479-cb6d-4e9e-9917-4b99d510653a],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-157[topN_[euclid_cdo_live_hourly]_14a9ad98-f3a5-407e-b183-f953f9f93df1],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-154[topN_[euclid_cdo_live_hourly]_aa2697cd-5e1c-419b-891b-1c1563dc07b2],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-167[topN_[euclid_cdo_live_hourly]_848e3e21-5da2-4449-a96f-9a79e8581cfa],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-183[topN_[euclid_cdo_live_hourly]_2b2727fa-bc35-455e-b947-bb90463965bb],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-160[topN_[euclid_cdo_live_hourly]_b3f4badf-f5da-4103-820e-37d97afba7d5],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-170[topN_[euclid_cdo_live_hourly]_714624b8-fd38-46a3-8044-17cb91f54acd],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-164[topN_[euclid_cdo_live_hourly]_1fd07432-53bd-433b-9cda-714660401bcc],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-156[topN_[euclid_cdo_live_hourly]_cf25e506-acd1-4979-bfe4-07c0b67b5547],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-158[topN_[euclid_cdo_live_hourly]_e3c5d7c0-0dc1-4098-a8cb-2d0ee3c04a4f],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-176[timeseries_[euclid_cdo_live_hourly]_7755967f-6778-4064-8a23-0b3bdb8742ac],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-172[topN_[euclid_cdo_live_hourly]_c20848de-6dde-43ae-86e2-4cfe54f8f126],5,main]
2020-06-11T08:27:27,920 ERROR [Thread-43] org.apache.druid.server.initialization.jetty.JettyServerModule - Jetty lifecycle event failed [class org.eclipse.jetty.server.Server]
java.util.concurrent.TimeoutException: null
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:129) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:181) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.server.Server.doStop(Server.java:439) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) [jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.apache.druid.server.initialization.jetty.JettyServerModule$2.stop(JettyServerModule.java:450) [druid-server-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:368) [druid-core-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.lifecycle.Lifecycle$1.run(Lifecycle.java:401) [druid-core-0.18.1.jar:0.18.1]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
2020-06-11T08:27:27,920 WARN [Thread-43] org.apache.druid.server.initialization.jetty.JettyServerModule - Unable to stop Jetty server.
java.util.concurrent.TimeoutException: null
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:129) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:181) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.server.Server.doStop(Server.java:439) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.apache.druid.server.initialization.jetty.JettyServerModule$2.stop(JettyServerModule.java:450) [druid-server-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:368) [druid-core-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.lifecycle.Lifecycle$1.run(Lifecycle.java:401) [druid-core-0.18.1.jar:0.18.1]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
2020-06-11T08:27:27,920 INFO [Thread-43] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2020-06-11T08:27:27,924 INFO [Thread-43] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView is stopping.
2020-06-11T08:27:27,924 INFO [Thread-43] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView Stopped.
2020-06-11T08:27:27,925 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2020-06-11T08:27:27,937 INFO [Thread-43] org.apache.druid.guice.LifecycleForkJoinPoolProvider - Shutting down ForkJoinPool [org.apache.druid.guice.LifecycleForkJoinPoolProvider@35bd2eec]
```
@gianm @sixtus Any idea how we can resolve this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org