You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/10/29 06:27:23 UTC
[GitHub] [hive] cnauroth opened a new pull request, #3713: HIVE-26677: Constrain available processors to Jetty during test runs …
cnauroth opened a new pull request, #3713:
URL: https://github.com/apache/hive/pull/3713
…to prevent thread exhaustion.
### What changes were proposed in this pull request?
As described during a [release candidate vote](https://lists.apache.org/thread/8qjf7x9t9v09d79hlzh712ls4zthdwrh):
[HIVE-24484](https://issues.apache.org/jira/browse/HIVE-24484) introduced a change to limit `hive.server2.webui.max.threads` to 4. Jetty enforces thread leasing to warn or abort if there aren't enough threads available [1]. During startup, it attempts to lease a thread per NIO selector [2]. By default, the number of NIO selectors to use is determined based on available CPUs [3]. This is mostly a passthrough to `Runtime.availableProcessors()` [4]. In my case, running on a machine with 16 CPUs, this ended up creating more than 4 selectors, therefore requiring more than 4 threads and violating the lease check. I was able to work around this by passing the `JETTY_AVAILABLE_PROCESSORS` system property to constrain the number of CPUs available to Jetty.
Since we are intentionally constraining the pool to 4 threads during itests, let's also limit `JETTY_AVAILABLE_PROCESSORS` in `maven.test.jvm.args` of the root pom.xml, so that others don't run into this problem later.
[1] https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ThreadPoolBudget.java#L165
[2] https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-io/src/main/java/org/eclipse/jetty/io/SelectorManager.java#L255
[3] https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-io/src/main/java/org/eclipse/jetty/io/SelectorManager.java#L79
[4] https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-util/src/main/java/org/eclipse/jetty/util/ProcessorUtils.java#L45
### Why are the changes needed?
This prevents spurious failures during itests, which could especially be a nuisance during release candidate verification.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
During the release candidate vote, I confirmed that this change fixed itests in my environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta merged pull request #3713: HIVE-26677: Constrain available processors to Jetty during test runs …
Posted by GitBox <gi...@apache.org>.
szlta merged PR #3713:
URL: https://github.com/apache/hive/pull/3713
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3713: HIVE-26677: Constrain available processors to Jetty during test runs …
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3713:
URL: https://github.com/apache/hive/pull/3713#issuecomment-1295755537
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3713)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3713&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3713&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3713&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3713&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3713&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3713&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3713&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3713&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3713&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3713&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3713&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3713&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3713&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3713&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org