You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/12/29 02:07:05 UTC
[GitHub] [hbase] cuibo01 opened a new pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
cuibo01 opened a new pull request #2824:
URL: https://github.com/apache/hbase/pull/2824
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752316984
> > I guess the problem is that, since you set ulimit to a very low value, once there is no new thred allowed, the 'kill' command itself can not be executed either...
>
> if this, in hbase, the 'kill' command itself can not be executed either
Yes, I think this is the root problem here. We need to find a way to kill the hbase process if OOME happens, not only for TimeoutExecutor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752314346
> I guess the problem is that, since you set ulimit to a very low value, once there is no new thred allowed, the 'kill' command itself can not be executed either...
but I can "kill -9" it manually...
steps:
1. ulimit -u 100
2. hbase com.xxxx.xxx.xxx.Test &
3. Wait for "unable to create new native thread" to be printed.
4. kill -9 pid
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-751947914
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 14s | master passed |
| +1 :green_heart: | compile | 1m 27s | master passed |
| +1 :green_heart: | shadedjars | 7m 4s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 2s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 5s | the patch passed |
| +1 :green_heart: | compile | 1m 33s | the patch passed |
| +1 :green_heart: | javac | 1m 33s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 50s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 44s | hbase-procedure in the patch passed. |
| +1 :green_heart: | unit | 146m 30s | hbase-server in the patch passed. |
| | | 178m 44s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 2474440aa7ed 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d963342f8a |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/1/testReport/ |
| Max. process+thread count | 3848 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752313202
I guess the problem is that, since you set ulimit to a very low value, once there is no new thred allowed, the 'kill' command itself can not be executed either...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752313702
https://www.codementor.io/@suryab/outofmemoryerror-related-jvm-arguments-w6e4vgipt
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
saintstack commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-753517691
I like how this PR is coming along.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
saintstack commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752809514
Agree. If OOME, process should go down... die. We can't keep going. It is a fatal condition.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 edited a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 edited a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752300485
> So we need to find out why this opinion does not work first? OOME can happen at any place and break the assumptions in code, there is no way to fix it cleanly just by adding more checks in code...
ok, mybe -XX:OnOutOfMemoryError=\\"kill -9 %p\\",
if there is no '\\', jvm executes kill instead of "kill -9 pid" when oom occurs.
and we also can enhance timeoutThread, it exits due to other reasons, master should be able to abort...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-751946331
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 40s | master passed |
| +1 :green_heart: | compile | 1m 17s | master passed |
| +1 :green_heart: | shadedjars | 6m 34s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 54s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 27s | the patch passed |
| +1 :green_heart: | compile | 1m 14s | the patch passed |
| +1 :green_heart: | javac | 1m 14s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 33s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 51s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 50s | hbase-procedure in the patch passed. |
| +1 :green_heart: | unit | 139m 4s | hbase-server in the patch passed. |
| | | 169m 3s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 43ff844eeba9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d963342f8a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/1/testReport/ |
| Max. process+thread count | 4695 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752817713
According to the communication with the maintenance, this problem that jvm does not exit exists in many OS components. No better solution is available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752833905
when throw "unable to create new native thread", the Thread does not run, uncaughtExceptionHandler is invalid.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 edited a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 edited a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752833905
when throw "unable to create new native thread", the Thread does not run, jvm does not call uncaughtExceptionHandler
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 edited a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 edited a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752816204
analyze and verify the java native code, some OMME can not be the JVM exit, like "java.lang.OutOfMemoryError: Direct buffer memory", "java.lang.OutOfMemoryError: unable to create new native thread", they dont call vmError#next_OnError_command.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-751927953
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 3s | master passed |
| +1 :green_heart: | checkstyle | 1m 27s | master passed |
| +1 :green_heart: | spotbugs | 2m 42s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 53s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 24s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 59s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| | | 45m 54s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 65cff0148f54 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d963342f8a |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752061346
> In general, if there is an OOME, we can not assume that our handler can still be executed correctly, as we do not know where is the OOME thrown out, it may break the assumptions in our code and cause unexpected behavior.
>
> IIRC, we have '-XX:OnOutOfMemoryError="kill -9 %p"' in our command line when start daemons? It does not work in your deployment?
>
> Thanks.
we can enhance OutOfMemoryError, if some users do not use '-XX:OnOutOfMemoryError="kill -9 %p"', or TimeOutThread exits due to other reasons, master should be able to abort...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-753584020
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 1s | master passed |
| +1 :green_heart: | compile | 1m 28s | master passed |
| +1 :green_heart: | shadedjars | 6m 40s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 3s | the patch passed |
| +1 :green_heart: | compile | 1m 27s | the patch passed |
| +1 :green_heart: | javac | 1m 27s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 40s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 58s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 43s | hbase-procedure in the patch passed. |
| +1 :green_heart: | unit | 130m 42s | hbase-server in the patch passed. |
| | | 162m 2s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux eea1d7f9dab5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2e21da7597 |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/4/testReport/ |
| Max. process+thread count | 4078 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752115741
Have we observed errors other than OOME which cause the TimeoutExecutorThread to hang there?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-753571327
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 18s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 41s | master passed |
| +1 :green_heart: | compile | 1m 16s | master passed |
| +1 :green_heart: | shadedjars | 6m 36s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 54s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 26s | the patch passed |
| +1 :green_heart: | compile | 1m 15s | the patch passed |
| +1 :green_heart: | javac | 1m 15s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 32s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 52s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 48s | hbase-procedure in the patch passed. |
| +1 :green_heart: | unit | 139m 14s | hbase-server in the patch passed. |
| | | 169m 55s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 85be80d4c01b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2e21da7597 |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/3/testReport/ |
| Max. process+thread count | 4493 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-753792850
thank all for reviewing the code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 edited a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 edited a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752295992
> In general, if there is an OOME, we can not assume that our handler can still be executed correctly, as we do not know where is the OOME thrown out, it may break the assumptions in our code and cause unexpected behavior.
>
> IIRC, we have '-XX:OnOutOfMemoryError="kill -9 %p"' in our command line when start daemons? It does not work in your deployment?
>
> Thanks.
and in our env, when OOME occurs, -XX:OnOutOfMemoryError="kill -9 %p" exists, but active master is not stopped.
https://github.com/apache/hbase/blob/c96fbf04077fa37555033f88939fdd69ac810b35/bin/hbase#L811
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752294308
> Have we observed errors other than OOME which cause the TimeoutExecutorThread to hang there?
now, we have not seen anything else except OOM, but we think there are other errors
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752300485
> So we need to find out why this opinion does not work first? OOME can happen at any place and break the assumptions in code, there is no way to fix it cleanly just by adding more checks in code...
ok, mybe -XX:OnOutOfMemoryError=\"kill -9 %p\",
if there is no '\', jvm executes kill instead of "kill -9 pid" when oom occurs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752316659
> I guess the problem is that, since you set ulimit to a very low value, once there is no new thred allowed, the 'kill' command itself can not be executed either...
if this, in hbase, the 'kill' command itself can not be executed either
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-753584637
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 30s | master passed |
| +1 :green_heart: | compile | 1m 17s | master passed |
| +1 :green_heart: | shadedjars | 6m 37s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 53s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 26s | the patch passed |
| +1 :green_heart: | compile | 1m 17s | the patch passed |
| +1 :green_heart: | javac | 1m 17s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 35s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 49s | hbase-procedure in the patch passed. |
| +1 :green_heart: | unit | 138m 44s | hbase-server in the patch passed. |
| | | 168m 40s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 3c3e065def05 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2e21da7597 |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/4/testReport/ |
| Max. process+thread count | 4599 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752312107
So the problem is OnOutOfMemoryError flag does not work for "Unable to create new native thread"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752316846
> Could you please try other options to see if it works?
i will continue to analyze. thank you @Apache9
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-753562412
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 54s | master passed |
| +1 :green_heart: | checkstyle | 1m 24s | master passed |
| +1 :green_heart: | spotbugs | 2m 42s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 42s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 22s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 51s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 3s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. |
| | | 43m 53s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 2e97a08b9c6d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2e21da7597 |
| Max. process+thread count | 94 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/3/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 removed a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 removed a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752902834
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 edited a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 edited a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752300485
> So we need to find out why this opinion does not work first? OOME can happen at any place and break the assumptions in code, there is no way to fix it cleanly just by adding more checks in code...
ok, mybe -XX:OnOutOfMemoryError=\"kill -9 %p\",
if there is no '\\', jvm executes kill instead of "kill -9 pid" when oom occurs.
and we also can enhance timeoutThread, it exits due to other reasons, master should be able to abort...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] pankaj72981 merged pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
pankaj72981 merged pull request #2824:
URL: https://github.com/apache/hbase/pull/2824
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752894540
> when throw "unable to create new native thread", the Thread does not run, jvm does not call uncaughtExceptionHandler
I do not get your point here. The current PR is to catch Throwable inside the run method of the TimeoutExecutor, which means it has already been started? Did I miss something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 edited a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 edited a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752816204
analyze and verify the java native code, some OMME can not cause the JVM to exit, like "java.lang.OutOfMemoryError: Direct buffer memory", "java.lang.OutOfMemoryError: unable to create new native thread", they dont call vmError#next_OnError_command.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752903200
> > > when throw "unable to create new native thread", the Thread does not run, jvm does not call uncaughtExceptionHandler
> >
> >
> > I do not get your point here. The current PR is to catch Throwable inside the run method of the TimeoutExecutor, which means it has already been started? Did I miss something?
>
> TimeoutExecutor is ok, the new thread is created in the threadPool and fails to be created, the threadPool passed the OOME to the TimeoutExecutor#run
So I suggest you to call setUncaughtExceptionHandler on TimeoutExecutor...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752983825
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 27s | master passed |
| +1 :green_heart: | compile | 1m 36s | master passed |
| +1 :green_heart: | shadedjars | 7m 6s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 0s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 18s | the patch passed |
| +1 :green_heart: | compile | 1m 33s | the patch passed |
| +1 :green_heart: | javac | 1m 33s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 19s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 50s | hbase-procedure in the patch passed. |
| -1 :x: | unit | 7m 42s | hbase-server in the patch failed. |
| | | 40m 44s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 204681e18375 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / fbf00f9c28 |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/2/testReport/ |
| Max. process+thread count | 753 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] pankaj72981 commented on a change in pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
pankaj72981 commented on a change in pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#discussion_r549604769
##########
File path: hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
##########
@@ -306,15 +315,22 @@ public TimeoutExecutorThread() {
@Override
public void run() {
while (running.get()) {
- final DelayedWithTimeout task = DelayedUtil.takeWithoutInterrupt(queue);
- if (task == null || task == DelayedUtil.DELAYED_POISON) {
- // the executor may be shutting down, and the task is just the shutdown request
- continue;
- }
- if (task instanceof DelayedTask) {
- threadPool.execute(((DelayedTask) task).getObject());
- } else {
- ((BufferNode) task).dispatch();
+ try {
+ final DelayedWithTimeout task = DelayedUtil.takeWithoutInterrupt(queue);
+ if (task == null || task == DelayedUtil.DELAYED_POISON) {
+ // the executor may be shutting down, and the task is just the shutdown request
+ continue;
+ }
+ if (task instanceof DelayedTask) {
+ threadPool.execute(((DelayedTask) task).getObject());
+ } else {
+ ((BufferNode) task).dispatch();
+ }
+ } catch (Throwable e) {
+ LOG.error("Caught error", e);
+ if (timeoutExecutorError != null) {
Review comment:
Should we abort for all exception or only for OOM?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752816204
analyze and verify the java native code, some OMME do not cause the JVM to exit, like "java.lang.OutOfMemoryError: Direct buffer memory", "java.lang.OutOfMemoryError: unable to create new native thread", they dont call vmError#next_OnError_command.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752908828
> So I suggest you to call setUncaughtExceptionHandler on TimeoutExecutor...
got it...thank you @Apache9
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752986444
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 5m 43s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 30s | master passed |
| +1 :green_heart: | checkstyle | 1m 36s | master passed |
| +1 :green_heart: | spotbugs | 3m 6s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 19s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 35s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 20m 13s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 47s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 52m 35s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 7cb487035689 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / fbf00f9c28 |
| Max. process+thread count | 94 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/2/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752902535
> > when throw "unable to create new native thread", the Thread does not run, jvm does not call uncaughtExceptionHandler
>
> I do not get your point here. The current PR is to catch Throwable inside the run method of the TimeoutExecutor, which means it has already been started? Did I miss something?
TimeoutExecutor is ok, the new thread is created in the threadPool and fails to be created, the threadPool passed the OOME to the TimeoutExecutor#run
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752902834
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 edited a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 edited a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752061346
> In general, if there is an OOME, we can not assume that our handler can still be executed correctly, as we do not know where is the OOME thrown out, it may break the assumptions in our code and cause unexpected behavior.
>
> IIRC, we have '-XX:OnOutOfMemoryError="kill -9 %p"' in our command line when start daemons? It does not work in your deployment?
>
> Thanks.
we can enhance OutOfMemoryError, if some users do not use '-XX:OnOutOfMemoryError="kill -9 %p"', or TimeOutThread exits due to other reasons, master should be able to abort...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-753574557
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 49s | master passed |
| +1 :green_heart: | checkstyle | 1m 19s | master passed |
| +1 :green_heart: | spotbugs | 2m 33s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 26s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 17s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 29s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 50s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 41m 56s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux daf28bc2bf60 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2e21da7597 |
| Max. process+thread count | 94 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/4/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752297484
> > In general, if there is an OOME, we can not assume that our handler can still be executed correctly, as we do not know where is the OOME thrown out, it may break the assumptions in our code and cause unexpected behavior.
> > IIRC, we have '-XX:OnOutOfMemoryError="kill -9 %p"' in our command line when start daemons? It does not work in your deployment?
> > Thanks.
>
> and in our env, when OOME occurs, -XX:OnOutOfMemoryError="kill -9 %p" exists, but active master is not stopped.
> https://github.com/apache/hbase/blob/c96fbf04077fa37555033f88939fdd69ac810b35/bin/hbase#L811
So we need to find out why this opinion does not work first? OOME can happen at any place and break the assumptions in code, there is no way to fix it cleanly just by adding more checks in code...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 removed a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 removed a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752817713
According to the communication with the maintenance, this problem that jvm does not exit exists in many OS components. No better solution is available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752819455
if we don't have a better way to solve the issue or circumvent the issue, we can fix timeoutThread first, and at least the Proc is correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752310671
i tested unable to "create new native thread" by "ulimit -u 100",
but jvm didn't stop..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 edited a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 edited a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752310671
i tested "unable to create new native thread" by "ulimit -u 100",
but jvm didn't stop..
i tested '-XX:OnOutOfMemoryError="kill -9"' %p or '-XX:+ExitOnOutOfMemoryError' or '-XX:+CrashOnOutOfMemoryError'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-753025292
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 23s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 2s | master passed |
| +1 :green_heart: | compile | 1m 17s | master passed |
| +1 :green_heart: | shadedjars | 7m 10s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 54s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 55s | the patch passed |
| +1 :green_heart: | compile | 1m 17s | the patch passed |
| +1 :green_heart: | javac | 1m 17s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 5s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 51s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 55s | hbase-procedure in the patch passed. |
| +1 :green_heart: | unit | 227m 20s | hbase-server in the patch passed. |
| | | 259m 47s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 11abd3b1dd93 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / fbf00f9c28 |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/2/testReport/ |
| Max. process+thread count | 3245 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on a change in pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on a change in pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#discussion_r549684381
##########
File path: hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
##########
@@ -306,15 +315,22 @@ public TimeoutExecutorThread() {
@Override
public void run() {
while (running.get()) {
- final DelayedWithTimeout task = DelayedUtil.takeWithoutInterrupt(queue);
- if (task == null || task == DelayedUtil.DELAYED_POISON) {
- // the executor may be shutting down, and the task is just the shutdown request
- continue;
- }
- if (task instanceof DelayedTask) {
- threadPool.execute(((DelayedTask) task).getObject());
- } else {
- ((BufferNode) task).dispatch();
+ try {
+ final DelayedWithTimeout task = DelayedUtil.takeWithoutInterrupt(queue);
+ if (task == null || task == DelayedUtil.DELAYED_POISON) {
+ // the executor may be shutting down, and the task is just the shutdown request
+ continue;
+ }
+ if (task instanceof DelayedTask) {
+ threadPool.execute(((DelayedTask) task).getObject());
+ } else {
+ ((BufferNode) task).dispatch();
+ }
+ } catch (Throwable e) {
+ LOG.error("Caught error", e);
+ if (timeoutExecutorError != null) {
Review comment:
yes. whatever the exception, the thread exits...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-751997962
In general, if there is an OOME, we can not assume that our handler can still be executed correctly, as we do not know where is the OOME thrown out, it may break the assumptions in our code and cause unexpected behavior.
IIRC, we have '-XX:OnOutOfMemoryError="kill -9 %p"' in our command line when start daemons? It does not work in your deployment?
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#discussion_r550775686
##########
File path: hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
##########
@@ -106,6 +106,10 @@ public boolean start() {
return true;
}
+ protected void setUncaughtExceptionHandler(UncaughtExceptionHandler eh) {
Review comment:
Better name it something else? Like setTimeoutExecutorUncaughtExceptionHandler? As we have a method all getUncaugtExceptionHandler in this class, for getting uncaughtExceptionHandler for the thread pool.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752878534
According to the JVM specifications, the subsequent actions of OOM are "undefined behaviors", that is, after OOM occurs, it is uncertain what must be or will not be done.
https://stackoverflow.com/questions/7606953/behavior-of-a-java-process-in-case-of-outofmemoryerror
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 edited a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 edited a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752300485
> So we need to find out why this opinion does not work first? OOME can happen at any place and break the assumptions in code, there is no way to fix it cleanly just by adding more checks in code...
ok, mybe -XX:OnOutOfMemoryError=\"kill -9 %p\",
if there is no '\', jvm executes kill instead of "kill -9 pid" when oom occurs.
and we also can enhance timeoutThread, it exits due to other reasons, master should be able to abort...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752295992
> In general, if there is an OOME, we can not assume that our handler can still be executed correctly, as we do not know where is the OOME thrown out, it may break the assumptions in our code and cause unexpected behavior.
>
> IIRC, we have '-XX:OnOutOfMemoryError="kill -9 %p"' in our command line when start daemons? It does not work in your deployment?
>
> Thanks.
and in our env, -XX:OnOutOfMemoryError="kill -9 %p" exists...
https://github.com/apache/hbase/blob/c96fbf04077fa37555033f88939fdd69ac810b35/bin/hbase#L811
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 edited a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 edited a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752310671
i tested "unable to create new native thread" by "ulimit -u 100",
but jvm didn't stop..
i tested '-XX:OnOutOfMemoryError="kill -9"' %p or '-XX:+ExitOnOutOfMemoryError' or '-XX:+CrashOnOutOfMemoryError'
OOME has 8 different kinds, i guess '-XX:OnOutOfMemoryError="kill -9"' doesn't work for all OOME
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: GC Overhead limit exceeded
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
java.lang.OutOfMemoryError: Permgen space
java.lang.OutOfMemoryError: Metaspace
java.lang.OutOfMemoryError: Unable to create new native thread
java.lang.OutOfMemoryError: Kill process or sacrifice child
java.lang.OutOfMemoryError: reason stack_trace_with_native_method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cuibo01 edited a comment on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
cuibo01 edited a comment on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752295992
> In general, if there is an OOME, we can not assume that our handler can still be executed correctly, as we do not know where is the OOME thrown out, it may break the assumptions in our code and cause unexpected behavior.
>
> IIRC, we have '-XX:OnOutOfMemoryError="kill -9 %p"' in our command line when start daemons? It does not work in your deployment?
>
> Thanks.
and in our env, when OOME occurs, -XX:OnOutOfMemoryError="kill -9 %p" exists...
https://github.com/apache/hbase/blob/c96fbf04077fa37555033f88939fdd69ac810b35/bin/hbase#L811
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752314818
Could you please try other options to see if it works?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-752825371
> if we don't have a better way to solve the issue or circumvent the issue, we can fix timeoutThread first, and at least the Proc is correct.
OK. Then since TimeoutExecutor is just a Thread, I suggest we use uncaughtExceptionHandler to deal with this? If we hit an uncaught exception, we abort master. It will be cleaner than the current solution.
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2824: HBASE-25447 remoteProc is suspended due to OOM ERROR
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2824:
URL: https://github.com/apache/hbase/pull/2824#issuecomment-753570644
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 57s | master passed |
| +1 :green_heart: | compile | 1m 29s | master passed |
| +1 :green_heart: | shadedjars | 6m 43s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 58s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 8s | the patch passed |
| +1 :green_heart: | compile | 1m 28s | the patch passed |
| +1 :green_heart: | javac | 1m 28s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 42s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 58s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 42s | hbase-procedure in the patch passed. |
| +1 :green_heart: | unit | 131m 6s | hbase-server in the patch passed. |
| | | 162m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2824 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 2f63615f31cd 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2e21da7597 |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/3/testReport/ |
| Max. process+thread count | 3938 (vs. ulimit of 30000) |
| modules | C: hbase-procedure hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2824/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org