You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by MitchelLabonte <gi...@git.apache.org> on 2017/12/12 15:28:55 UTC
[GitHub] drill pull request #1069: DRILL-5994 Added webserver maxThreads configuratio...
GitHub user MitchelLabonte opened a pull request:
https://github.com/apache/drill/pull/1069
DRILL-5994 Added webserver maxThreads configuration option to enable …
…launching on a machine with more than 200 cores
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MitchelLabonte/drill DRILL-5994
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/1069.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1069
----
commit af4e99f16be4081f151024688d2fd5cd306b219b
Author: mitchel <mi...@hotmail.com>
Date: 2017-12-12T15:27:19Z
DRILL-5994 Added webserver maxThreads configuration option to enable launching on a machine with more than 200 cores
----
---
[GitHub] drill issue #1069: DRILL-5994 Added webserver maxThreads configuration optio...
Posted by vrozov <gi...@git.apache.org>.
Github user vrozov commented on the issue:
https://github.com/apache/drill/pull/1069
@MitchelLabonte My proposal is to limit the number of acceptors instead of allowing to increase the max number of threads in the connection pool. If the number of acceptors is limited, the number of "needed" threads will be limited too and the exception will not be raised.
---
[GitHub] drill issue #1069: DRILL-5994 Added webserver maxThreads configuration optio...
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1069
+1, LGTM.
---
[GitHub] drill pull request #1069: DRILL-5994 Added webserver maxThreads configuratio...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/1069
---
[GitHub] drill issue #1069: DRILL-5994 Added webserver maxThreads configuration optio...
Posted by vrozov <gi...@git.apache.org>.
Github user vrozov commented on the issue:
https://github.com/apache/drill/pull/1069
@MitchelLabonte @arina-ielchiieva I don't think that Drill needs that many threads/acceptors to handle HTTP(s) requests as it is not a real web (REST API) server. For proper resource utilization, it will be better to limit the number of acceptors to a small value (let's say 2 or 4 by default) instead of the current default that uses a number of available processors and can be huge on machines with lots of cores (32 or more). @paul-rogers What is your take on this?
---
[GitHub] drill pull request #1069: DRILL-5994 Added webserver maxThreads configuratio...
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1069#discussion_r156403329
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -377,7 +377,9 @@ drill.exec: {
# refresh time.
grace_period_ms : 0,
//port hunting for drillbits. Enabled only for testing purposes.
- port_hunt : false
+ port_hunt : false,
+ // Max threads of embedded Jetty
+ webserver.threadpool_max: 200
--- End diff --
Please add underscores: `web_server.thread_pool_max`
---
[GitHub] drill issue #1069: DRILL-5994 Added webserver maxThreads configuration optio...
Posted by MitchelLabonte <gi...@git.apache.org>.
Github user MitchelLabonte commented on the issue:
https://github.com/apache/drill/pull/1069
@vrozov the Web server definitely doesn't need this much, but Drill won't launch on a machine with more than 200 cores and throw an exception:
Exception in thread "main" org.apache.drill.exec.exception.DrillStartupException: Failure
during initial startup of Drillbit:
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:313)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:289)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
> Caused by: java.lang.IllegalStateException: Insufficient max threads in ThreadPool: max=200
< needed=206
> at org.eclipse.jetty.server.Server.doStart(Server.java:321)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at org.eclipse.drill.exec.server.rest.WebServer.start(WebServer.java:197)
> at org.eclipse.drill.exec.server.Drillbit.run(Drillbit.java:140)
> at org.eclipse.drill.exec.server.Drillbit.start(Drillbit.java:309)
> ... 2 more
The configuration is there to have an option to not hit this exception on startup on a machine with more than 200 cores.
---