You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hugegraph.apache.org by "pallavithegod (via GitHub)" <gi...@apache.org> on 2026/10/16 10:20:05 UTC
[PR] fix(dist): diagnose jvm thread creation failures during startup [hugegraph]
pallavithegod opened a new pull request, #3360:
URL: https://github.com/apache/hugegraph/pull/3360
## Main Changes (主要改动)
Close #3072.
- Root cause: Docker startup failures caused by denied JVM thread creation leave their useful `pthread_create`/EPERM explanation in crash files or the server log, while the entrypoint reports only startup failure.
- Fix details: inspect crash files created during the current startup and newly written server log bytes when init-store or server startup fails. Print seccomp and process-limit troubleshooting hints for EPERM, and memory/process-limit guidance for generic GC-thread creation failures. Preserve exit status, ignore stale diagnostics, and leave runtime security settings unchanged. Document the diagnostic flags and normal-operation guidance in the Docker README.
- Tests: extend `test-docker-entrypoint-process.sh` with current/stale crash and log cases, generic GC-thread failures, and init-store failure status. The entrypoint configuration/process fixture passes, as does `HugeSecurityManagerTest` through the distribution Maven reactor.
## Verifying these changes (验证改动)
- [x] Added/updated unit tests in `test-docker-entrypoint-process.sh`
- [x] Verified `mvn checkstyle:check` and `mvn apache-rat:check` pass locally
Java 17 / Maven 3.9.11 validation:
- `mvn apache-rat:check`
- `mvn editorconfig:format`
- `mvn clean compile -Dmaven.javadoc.skip=true -pl hugegraph-server/hugegraph-dist -am`
- `mvn test-compile checkstyle:check -pl hugegraph-server/hugegraph-dist -am -Dcheckstyle.config.location=/workspace/style/checkstyle.xml`
- `mvn test -pl hugegraph-server/hugegraph-dist -am -Dtest=HugeSecurityManagerTest -Dsurefire.failIfNoSpecifiedTests=false`
- `bash hugegraph-server/hugegraph-dist/docker/docker-entrypoint-test.sh` (includes process and partition-wait fixtures)
The explicit Checkstyle configuration applies the repository's rules to the distributed modules included by the reactor. No Java test class is needed for the shell-only change; the new regression cases run in the existing shell/Python process fixture.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(dist): diagnose jvm thread creation failures during startup [hugegraph]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #3360:
URL: https://github.com/apache/hugegraph/pull/3360#discussion_r4306904119
##########
hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh:
##########
@@ -16,6 +16,60 @@
# limitations under the License.
#
+function record_jvm_startup_diagnostics() {
+ JVM_STARTUP_TIME=$(date -u +'%Y-%m-%d %H:%M:%S.%N UTC')
+ JVM_STARTUP_LOG_OFFSET=$(stat -c %s ./logs/hugegraph-server.log 2>/dev/null || echo 0)
+ JVM_STARTUP_STDOUT_OFFSET=$(stat -c %s ./logs/hugegraph-server-stdout.log 2>/dev/null || echo 0)
+}
+
+function print_jvm_thread_hint() {
+ local kind
+ kind=$(awk '
+ /pthread_create.*\(EPERM\)|pthread_create.*Operation not permitted/ { denied = 1 }
+ /[Tt]hread.*Operation not permitted/ { denied = 1 }
Review Comment:
Minor: This rule matches any line that contains "thread" before "Operation not permitted", not only thread-creation failures. Two kinds of startup output have that shape. First, the JVM's handler for an uncaught exception prints `Exception in thread "main" ...`, and `HugeGraphServer.main` rethrows startup failures. With `STDOUT_MODE=false` that line lands in `logs/hugegraph-server-stdout.log`, which is now scanned first. Second, the log4j pattern `[%t]` puts thread names such as `pool-3-thread-1` into every `hugegraph-server.log` line, and that file is scanned in the default image. So a startup that fails on a file EPERM (a bind mount refusing chmod or chown, for example) is reported as `JVM thread creation was denied (EPERM)` with seccomp advice. I fed both shapes to this head's `print_jvm_thread_hint`, and both printed the denied hint:
```
Exception in thread "main" java.nio.file.FileSystemException: /hugegraph-server/rocksdb-data/m/LOCK: Operation not permitted
2026-10-18 11:36:00 [pool-3-thread-1] [ERROR] o.a.h.b.s.r.RocksDBStore - Failed to open RocksDB: IO error: lock: Operation not permitted
```
Line 28 already covers both HotSpot spellings (`pthread_create failed (EPERM)` and `pthread_create ... Operation not permitted`), so this rule adds only these false matches.
Requested change: drop this rule. If it targets a specific JVM message that does not mention `pthread_create`, narrow it to that message, for example `Failed to start thread`. Then add a negative fixture: an unrelated `Operation not permitted` line in the new stdout or server-log bytes, asserting that the denied hint is not printed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(dist): diagnose jvm thread creation failures during startup [hugegraph]
Posted by "pallavithegod (via GitHub)" <gi...@apache.org>.
pallavithegod commented on code in PR #3360:
URL: https://github.com/apache/hugegraph/pull/3360#discussion_r4306710284
##########
hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh:
##########
@@ -16,6 +16,51 @@
# limitations under the License.
#
+function record_jvm_startup_diagnostics() {
+ JVM_STARTUP_TIME=$(date -u +'%Y-%m-%d %H:%M:%S.%N UTC')
+ JVM_STARTUP_LOG_OFFSET=$(stat -c %s ./logs/hugegraph-server.log 2>/dev/null || echo 0)
+}
+
+function print_jvm_thread_hint() {
+ local kind
+ kind=$(awk '
+ /pthread_create.*\(EPERM\)|pthread_create.*Operation not permitted/ { denied = 1 }
+ /[Tt]hread.*Operation not permitted/ { denied = 1 }
+ /Cannot create worker GC thread/ { resources = 1 }
+ END { if (denied) print "denied"; else if (resources) print "resources" }
+ ')
+ case "$kind" in
+ denied)
+ echo 'ERROR: JVM thread creation was denied (EPERM). Check the Docker host seccomp profile and security policy.' >&2
+ echo 'To diagnose seccomp restrictions, retry with --security-opt seccomp=unconfined; use a compatible host/runtime profile for normal operation.' >&2
+ ;;
+ resources)
+ echo 'ERROR: JVM could not create a GC worker thread. Possible causes include seccomp denial, insufficient memory, or process/thread limits.' >&2
+ echo 'Look for pthread_create failed (EPERM) above in the container log; if present, check the host seccomp profile and security policy.' >&2
Review Comment:
Fixed in ae3e701cc. Startup records the stdout-file offset and scans its new bytes before generic crash headers; the fallback hint also names that file. Added STDOUT_MODE=false cases with and without a crash file, stale stdout, and a truncated stdout file. The complete entrypoint fixture, focused Maven tests, clean compilation, RAT, and repository Checkstyle pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(dist): diagnose jvm thread creation failures during startup [hugegraph]
Posted by "pallavithegod (via GitHub)" <gi...@apache.org>.
pallavithegod commented on code in PR #3360:
URL: https://github.com/apache/hugegraph/pull/3360#discussion_r4316759622
##########
hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh:
##########
@@ -16,6 +16,60 @@
# limitations under the License.
#
+function record_jvm_startup_diagnostics() {
+ JVM_STARTUP_TIME=$(date -u +'%Y-%m-%d %H:%M:%S.%N UTC')
+ JVM_STARTUP_LOG_OFFSET=$(stat -c %s ./logs/hugegraph-server.log 2>/dev/null || echo 0)
+ JVM_STARTUP_STDOUT_OFFSET=$(stat -c %s ./logs/hugegraph-server-stdout.log 2>/dev/null || echo 0)
+}
+
+function print_jvm_thread_hint() {
+ local kind
+ kind=$(awk '
+ /pthread_create.*\(EPERM\)|pthread_create.*Operation not permitted/ { denied = 1 }
+ /[Tt]hread.*Operation not permitted/ { denied = 1 }
Review Comment:
Removed the broad thread-name rule in 67d1a37fd and added negative fixtures for filesystem EPERM in fresh stdout and server-log bytes. All 32 process cases, four diagnostic controls, the focused Maven test, compilation, RAT and repository Checkstyle pass; the PR also states the append/observed-shrink boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(dist): diagnose jvm thread creation failures during startup [hugegraph]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3360:
URL: https://github.com/apache/hugegraph/pull/3360#issuecomment-6241774798
## [Codecov](https://app.codecov.io/gh/apache/hugegraph/pull/3360?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 42.15%. Comparing base ([`92af2f8`](https://app.codecov.io/gh/apache/hugegraph/commit/92af2f89c320c3e6fa7c10765a061f3edadea319?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a24c32d`](https://app.codecov.io/gh/apache/hugegraph/commit/a24c32d1b3794b302859ecde97158b75de570f3f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 31 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3360 +/- ##
=============================================
- Coverage 54.31% 42.15% -12.16%
+ Complexity 2250 2 -2248
=============================================
Files 770 506 -264
Lines 67952 48579 -19373
Branches 9213 7022 -2191
=============================================
- Hits 36905 20478 -16427
+ Misses 27470 25719 -1751
+ Partials 3577 2382 -1195
```
| [Flag](https://app.codecov.io/gh/apache/hugegraph/pull/3360/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [server_rocksdb](https://app.codecov.io/gh/apache/hugegraph/pull/3360/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `42.15% <ø> (?)` | |
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hugegraph/pull/3360?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(dist): diagnose jvm thread creation failures during startup [hugegraph]
Posted by "pallavithegod (via GitHub)" <gi...@apache.org>.
pallavithegod commented on PR #3360:
URL: https://github.com/apache/hugegraph/pull/3360#issuecomment-6216196963
Thanks for reproducing this. I will use realistic JVM stdout/crash-file fixtures, make the generic crash hint cover the possible seccomp cause, and remove process-limit advice from the EPERM-specific hint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(dist): diagnose jvm thread creation failures during startup [hugegraph]
Posted by "pallavithegod (via GitHub)" <gi...@apache.org>.
pallavithegod commented on code in PR #3360:
URL: https://github.com/apache/hugegraph/pull/3360#discussion_r4295082061
##########
hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh:
##########
@@ -16,6 +16,50 @@
# limitations under the License.
#
+function record_jvm_startup_diagnostics() {
+ JVM_STARTUP_TIME=$(date -u +'%Y-%m-%d %H:%M:%S.%N UTC')
+ JVM_STARTUP_LOG_OFFSET=$(stat -c %s ./logs/hugegraph-server.log 2>/dev/null || echo 0)
+}
+
+function print_jvm_thread_hint() {
+ local kind
+ kind=$(awk '
+ /pthread_create.*\(EPERM\)|pthread_create.*Operation not permitted/ { denied = 1 }
+ /[Tt]hread.*Operation not permitted/ { denied = 1 }
+ /Cannot create worker GC thread/ { resources = 1 }
+ END { if (denied) print "denied"; else if (resources) print "resources" }
+ ')
+ case "$kind" in
+ denied)
+ echo 'ERROR: JVM thread creation was denied (EPERM). Check the Docker host seccomp profile and thread limits.' >&2
+ echo 'To diagnose seccomp restrictions, retry with --security-opt seccomp=unconfined; use a compatible host/runtime profile for normal operation.' >&2
+ echo 'If the process limit is too low, set --ulimit nproc=65535:65535.' >&2
+ ;;
+ resources)
+ echo 'ERROR: JVM could not create a GC worker thread. Check available memory and process/thread limits.' >&2
+ echo 'If the process limit is too low, set --ulimit nproc=65535:65535.' >&2
+ ;;
+ *) return 1 ;;
+ esac
+}
+
+function diagnose_jvm_startup_failure() {
+ local crash size offset
+ while IFS= read -r -d '' crash; do
+ if [[ -r "$crash" ]] && print_jvm_thread_hint < "$crash"; then
Review Comment:
Updated in a24c32d1b using the suggested generic-hint approach. A generic GC-thread crash now lists seccomp denial alongside memory and process limits and directs readers to the preceding JVM stdout for EPERM. The process fixtures now put the EPERM warning on stdout and only the generic header in the crash file, including init-store exit-status coverage. The entrypoint/process fixtures, RAT, repository Checkstyle, clean compilation, and HugeSecurityManagerTest pass.
##########
hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh:
##########
@@ -16,6 +16,50 @@
# limitations under the License.
#
+function record_jvm_startup_diagnostics() {
+ JVM_STARTUP_TIME=$(date -u +'%Y-%m-%d %H:%M:%S.%N UTC')
+ JVM_STARTUP_LOG_OFFSET=$(stat -c %s ./logs/hugegraph-server.log 2>/dev/null || echo 0)
+}
+
+function print_jvm_thread_hint() {
+ local kind
+ kind=$(awk '
+ /pthread_create.*\(EPERM\)|pthread_create.*Operation not permitted/ { denied = 1 }
+ /[Tt]hread.*Operation not permitted/ { denied = 1 }
+ /Cannot create worker GC thread/ { resources = 1 }
+ END { if (denied) print "denied"; else if (resources) print "resources" }
+ ')
+ case "$kind" in
+ denied)
+ echo 'ERROR: JVM thread creation was denied (EPERM). Check the Docker host seccomp profile and thread limits.' >&2
+ echo 'To diagnose seccomp restrictions, retry with --security-opt seccomp=unconfined; use a compatible host/runtime profile for normal operation.' >&2
+ echo 'If the process limit is too low, set --ulimit nproc=65535:65535.' >&2
Review Comment:
Updated in a24c32d1b. The EPERM-specific branch now refers only to seccomp/security policy, with no thread-limit or nproc advice. The README distinguishes EPERM from EAGAIN, and a focused assertion checks that the EPERM hint contains no process-limit advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(dist): diagnose jvm thread creation failures during startup [hugegraph]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #3360:
URL: https://github.com/apache/hugegraph/pull/3360#discussion_r4294281826
##########
hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh:
##########
@@ -16,6 +16,50 @@
# limitations under the License.
#
+function record_jvm_startup_diagnostics() {
+ JVM_STARTUP_TIME=$(date -u +'%Y-%m-%d %H:%M:%S.%N UTC')
+ JVM_STARTUP_LOG_OFFSET=$(stat -c %s ./logs/hugegraph-server.log 2>/dev/null || echo 0)
+}
+
+function print_jvm_thread_hint() {
+ local kind
+ kind=$(awk '
+ /pthread_create.*\(EPERM\)|pthread_create.*Operation not permitted/ { denied = 1 }
+ /[Tt]hread.*Operation not permitted/ { denied = 1 }
+ /Cannot create worker GC thread/ { resources = 1 }
+ END { if (denied) print "denied"; else if (resources) print "resources" }
+ ')
+ case "$kind" in
+ denied)
+ echo 'ERROR: JVM thread creation was denied (EPERM). Check the Docker host seccomp profile and thread limits.' >&2
+ echo 'To diagnose seccomp restrictions, retry with --security-opt seccomp=unconfined; use a compatible host/runtime profile for normal operation.' >&2
+ echo 'If the process limit is too low, set --ulimit nproc=65535:65535.' >&2
Review Comment:
Minor: The EPERM branch tells users to raise `--ulimit nproc`, and line 34 points them at thread limits too, but a process or thread limit cannot cause this error. When `RLIMIT_NPROC` or a cgroup `pids.max` is hit, `clone`/`pthread_create` fail with EAGAIN (pthread_create(3): "A system-imposed limit on the number of threads was encountered"), and the JVM then logs `pthread_create failed (EAGAIN)`. EPERM points at a seccomp or LSM denial, so the `nproc` advice sends people after a cause that cannot produce this message.
Requested change: drop the `nproc` line and the "thread limits" wording from the `denied` branch, keep them in the `resources` branch, and match the README paragraph to that.
##########
hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh:
##########
@@ -16,6 +16,50 @@
# limitations under the License.
#
+function record_jvm_startup_diagnostics() {
+ JVM_STARTUP_TIME=$(date -u +'%Y-%m-%d %H:%M:%S.%N UTC')
+ JVM_STARTUP_LOG_OFFSET=$(stat -c %s ./logs/hugegraph-server.log 2>/dev/null || echo 0)
+}
+
+function print_jvm_thread_hint() {
+ local kind
+ kind=$(awk '
+ /pthread_create.*\(EPERM\)|pthread_create.*Operation not permitted/ { denied = 1 }
+ /[Tt]hread.*Operation not permitted/ { denied = 1 }
+ /Cannot create worker GC thread/ { resources = 1 }
+ END { if (denied) print "denied"; else if (resources) print "resources" }
+ ')
+ case "$kind" in
+ denied)
+ echo 'ERROR: JVM thread creation was denied (EPERM). Check the Docker host seccomp profile and thread limits.' >&2
+ echo 'To diagnose seccomp restrictions, retry with --security-opt seccomp=unconfined; use a compatible host/runtime profile for normal operation.' >&2
+ echo 'If the process limit is too low, set --ulimit nproc=65535:65535.' >&2
+ ;;
+ resources)
+ echo 'ERROR: JVM could not create a GC worker thread. Check available memory and process/thread limits.' >&2
+ echo 'If the process limit is too low, set --ulimit nproc=65535:65535.' >&2
+ ;;
+ *) return 1 ;;
+ esac
+}
+
+function diagnose_jvm_startup_failure() {
+ local crash size offset
+ while IFS= read -r -d '' crash; do
+ if [[ -r "$crash" ]] && print_jvm_thread_hint < "$crash"; then
Review Comment:
Important: In the #3072 failure the crash file never contains the EPERM text, so this loop takes the `resources` branch and returns. Users get the memory hint and never see the seccomp one.
HotSpot prints `Failed to start thread "GC Thread#0" - pthread_create failed (EPERM)` as a unified-logging warning on its stdout. The `hs_err_pid*.log` file only carries the generic header. I reproduced it on Docker 29.5.3 with a seccomp profile that returns EPERM for `clone3`:
```
docker run --rm --security-opt seccomp=$d/seccomp.json -v $d/out:/out -w /out \
eclipse-temurin:17-jre-jammy java -XX:ErrorFile=/out/hs_err_pid%p.log -version
```
stdout had the EPERM warning, the same output as in the issue. `grep -E 'pthread_create|EPERM|Operation not permitted|Cannot create worker GC thread' hs_err_pid7.log` matched one line, `# Cannot create worker GC thread. Out of system resources.`
The log fallback below can't catch the EPERM line either. `init-store.sh` sends the JVM's stdout to the container log. `hugegraph-server.sh` sends it to container stdout under `STDOUT_MODE=true`, which both Dockerfiles set, and to `hugegraph-server-stdout.log` otherwise. Neither path writes `hugegraph-server.log`. And this loop returns before the log is read anyway.
The `early42-eperm-file` and `early42-eperm-log` fixtures pass only because they write the EPERM text into files a real JVM never puts it in. `early42-resource` already has the real crash-file shape, and it asserts that the seccomp hint is absent. To check this, I changed that fixture so the JVM substitute prints the real warning to stdout and writes the real `hs_err` header, then asserted the seccomp hint. The test failed with `ERROR: JVM could not create a GC worker thread. Check available memory and process/thread limits.`
Requested change: scan the failing JVM's own output. For example, tee `init-store.sh` and the launcher output to a temp file and run `print_jvm_thread_hint` on it. Or, if that is too invasive, make the `Cannot create worker GC thread` hint name both causes (seccomp denial and memory or limits) and tell users to look for `pthread_create failed (EPERM)` just above it in the container log. Either way, base the fixtures on the real JVM output (EPERM on stdout, generic text in `hs_err`), and update the README paragraph that says the crash files and Server log are checked for EPERM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(dist): diagnose jvm thread creation failures during startup [hugegraph]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #3360:
URL: https://github.com/apache/hugegraph/pull/3360#discussion_r4298177816
##########
hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh:
##########
@@ -16,6 +16,51 @@
# limitations under the License.
#
+function record_jvm_startup_diagnostics() {
+ JVM_STARTUP_TIME=$(date -u +'%Y-%m-%d %H:%M:%S.%N UTC')
+ JVM_STARTUP_LOG_OFFSET=$(stat -c %s ./logs/hugegraph-server.log 2>/dev/null || echo 0)
+}
+
+function print_jvm_thread_hint() {
+ local kind
+ kind=$(awk '
+ /pthread_create.*\(EPERM\)|pthread_create.*Operation not permitted/ { denied = 1 }
+ /[Tt]hread.*Operation not permitted/ { denied = 1 }
+ /Cannot create worker GC thread/ { resources = 1 }
+ END { if (denied) print "denied"; else if (resources) print "resources" }
+ ')
+ case "$kind" in
+ denied)
+ echo 'ERROR: JVM thread creation was denied (EPERM). Check the Docker host seccomp profile and security policy.' >&2
+ echo 'To diagnose seccomp restrictions, retry with --security-opt seccomp=unconfined; use a compatible host/runtime profile for normal operation.' >&2
+ ;;
+ resources)
+ echo 'ERROR: JVM could not create a GC worker thread. Possible causes include seccomp denial, insufficient memory, or process/thread limits.' >&2
+ echo 'Look for pthread_create failed (EPERM) above in the container log; if present, check the host seccomp profile and security policy.' >&2
Review Comment:
Minor: With `STDOUT_MODE=false` this hint sends users to the wrong place, and the EPERM-specific hint can never fire from the entrypoint.
`hugegraph-server.sh:374` appends the JVM's stdout and stderr to `logs/hugegraph-server-stdout.log` when `STDOUT_MODE` is not `true`. HotSpot's `pthread_create failed (EPERM)` warning then lands in that file, not in the container log that line 39 points to. The log scan below reads only `logs/hugegraph-server.log`, which holds log4j output and launcher errors and never receives HotSpot's stdout. It also runs only after the crash-file loop, which returns on the generic `Cannot create worker GC thread` header first.
I ran `record_jvm_startup_diagnostics`, wrote the real HotSpot EPERM warning to `logs/hugegraph-server-stdout.log` and the generic header to a new `logs/hs_err_pid42_<host>_<time>.log`, then called `diagnose_jvm_startup_failure` (Linux, this head's `util.sh`). It printed the generic hint with "Look for pthread_create failed (EPERM) above in the container log". With the crash file removed it printed nothing.
Requested change: record the size of `logs/hugegraph-server-stdout.log` in `record_jvm_startup_diagnostics` and scan its new bytes before the crash files, so the `denied` branch is reached when the warning is in that file. If you'd rather not, make line 39 name `logs/hugegraph-server-stdout.log` for the `STDOUT_MODE=false` case. Either way, add a fixture for that mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(dist): diagnose jvm thread creation failures during startup [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin merged PR #3360:
URL: https://github.com/apache/hugegraph/pull/3360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org