You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Enrico Olivelli <eo...@gmail.com> on 2019/09/24 14:36:13 UTC
OutOfMemory on Jenkins Slave H37
Hello,
I see this OutOfMemoryError on H37, can you please take a look ?
https://builds.apache.org/job/bookkeeper_precommit_replication_tests/1244/
hudson.plugins.git.GitException: Could not init
/home/jenkins/jenkins-slave/workspace/bookkeeper_precommit_replication_tests
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:813)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:605)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to H37
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:955)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor1084.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy135.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1810)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Error performing git command
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2051)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:811)
... 11 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at hudson.Proc$LocalProc.<init>(Proc.java:269)
at hudson.Proc$LocalProc.<init>(Proc.java:218)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:936)
at hudson.Launcher$ProcStarter.start(Launcher.java:455)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2038)
... 15 more
Enrico Olivelli
BookKeeper committer
RE: OutOfMemory on Jenkins Slave H37
Posted by Eric Barboni <sk...@apache.org>.
Hi,
I see also an oom on H40
https://builds.apache.org/view/M-R/view/NetBeans/job/netbeans-TLP/job/netbeans/job/release111/43/console
Best Regards
Eric
-----Message d'origine-----
De : Enrico Olivelli <eo...@gmail.com>
Envoyé : jeudi 26 septembre 2019 14:51
À : builds@apache.org
Objet : Re: OutOfMemory on Jenkins Slave H37
Hello again,
it seems that those machines are not very healthy this time I have problems on H41
The native memory error appears during the boot of the job, while maven is downloading the first set of dependencies in this case
The job dumps a lot of information about the environment, maybe it can be useful to track down the cause
https://builds.apache.org/job/bookkeeper_precommit_pullrequest_java11/543/console
[17.225s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
2019-09-26T12:42:48.579 [ERROR] unable to create native thread:
possibly out of memory or process/resource limits reached -> [Help 1]
Enrico
Il giorno mar 24 set 2019 alle ore 20:49 Chris Lambertus <cm...@apache.org> ha scritto:
> I rebooted the machine, but it didn’t come back up. Waiting on the
> colo to take a look at it.
>
>
> > On Sep 24, 2019, at 7:36 AM, Enrico Olivelli <eo...@gmail.com>
> wrote:
> >
> > Hello,
> > I see this OutOfMemoryError on H37, can you please take a look ?
> >
> >
> https://builds.apache.org/job/bookkeeper_precommit_replication_tests/1
> 244/
> >
> > hudson.plugins.git.GitException: Could not init
> >
> /home/jenkins/jenkins-slave/workspace/bookkeeper_precommit_replication
> _tests
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.
> java:813)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.
> java:605)
> > at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler
> $1.call(RemoteGitImpl.java:153)
> > at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler
> $1.call(RemoteGitImpl.java:146)
> > at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> > at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> > at hudson.remoting.Request$2.run(Request.java:369)
> > at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecuto
> rService.java:72)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> ava:1142)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> java:617)
> > at java.lang.Thread.run(Thread.java:748)
> > Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to H37
> > at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> > at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.jav
> a:357)
> > at hudson.remoting.Channel.call(Channel.java:955)
> > at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler
> .execute(RemoteGitImpl.java:146)
> > at
> > sun.reflect.GeneratedMethodAccessor1084.invoke(Unknown
> Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler
> .invoke(RemoteGitImpl.java:132)
> > at com.sun.proxy.$Proxy135.execute(Unknown Source)
> > at
> hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152)
> > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> > at hudson.scm.SCM.checkout(SCM.java:504)
> > at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> > at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(Abst
> ractBuild.java:574)
> > at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> > at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.ja
> va:499)
> > at hudson.model.Run.execute(Run.java:1810)
> > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> > at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> > at hudson.model.Executor.run(Executor.java:429)
> > Caused by: hudson.plugins.git.GitException: Error performing git command
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAP
> IImpl.java:2051)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAP
> IImpl.java:2010)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAP
> IImpl.java:2006)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPII
> mpl.java:1638)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.
> java:811)
> > ... 11 more
> > Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> > at java.lang.Thread.start0(Native Method)
> > at java.lang.Thread.start(Thread.java:717)
> > at hudson.Proc$LocalProc.<init>(Proc.java:269)
> > at hudson.Proc$LocalProc.<init>(Proc.java:218)
> > at hudson.Launcher$LocalLauncher.launch(Launcher.java:936)
> > at hudson.Launcher$ProcStarter.start(Launcher.java:455)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAP
> IImpl.java:2038)
> > ... 15 more
> >
> >
> >
> >
> > Enrico Olivelli
> >
> > BookKeeper committer
>
>
Re: OutOfMemory on Jenkins Slave H37
Posted by Enrico Olivelli <eo...@gmail.com>.
Hello again,
it seems that those machines are not very healthy this time I have problems
on H41
The native memory error appears during the boot of the job, while maven is
downloading the first set of dependencies in this case
The job dumps a lot of information about the environment, maybe it can be
useful to track down the cause
https://builds.apache.org/job/bookkeeper_precommit_pullrequest_java11/543/console
[17.225s][warning][os,thread] Failed to start thread - pthread_create
failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k,
detached.
2019-09-26T12:42:48.579 [ERROR] unable to create native thread:
possibly out of memory or process/resource limits reached -> [Help 1]
Enrico
Il giorno mar 24 set 2019 alle ore 20:49 Chris Lambertus <cm...@apache.org>
ha scritto:
> I rebooted the machine, but it didn’t come back up. Waiting on the colo to
> take a look at it.
>
>
> > On Sep 24, 2019, at 7:36 AM, Enrico Olivelli <eo...@gmail.com>
> wrote:
> >
> > Hello,
> > I see this OutOfMemoryError on H37, can you please take a look ?
> >
> >
> https://builds.apache.org/job/bookkeeper_precommit_replication_tests/1244/
> >
> > hudson.plugins.git.GitException: Could not init
> >
> /home/jenkins/jenkins-slave/workspace/bookkeeper_precommit_replication_tests
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:813)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:605)
> > at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
> > at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
> > at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> > at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> > at hudson.remoting.Request$2.run(Request.java:369)
> > at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:748)
> > Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to H37
> > at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> > at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> > at hudson.remoting.Channel.call(Channel.java:955)
> > at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
> > at sun.reflect.GeneratedMethodAccessor1084.invoke(Unknown
> Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
> > at com.sun.proxy.$Proxy135.execute(Unknown Source)
> > at
> hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152)
> > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> > at hudson.scm.SCM.checkout(SCM.java:504)
> > at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> > at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> > at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> > at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> > at hudson.model.Run.execute(Run.java:1810)
> > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> > at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> > at hudson.model.Executor.run(Executor.java:429)
> > Caused by: hudson.plugins.git.GitException: Error performing git command
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2051)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:811)
> > ... 11 more
> > Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> > at java.lang.Thread.start0(Native Method)
> > at java.lang.Thread.start(Thread.java:717)
> > at hudson.Proc$LocalProc.<init>(Proc.java:269)
> > at hudson.Proc$LocalProc.<init>(Proc.java:218)
> > at hudson.Launcher$LocalLauncher.launch(Launcher.java:936)
> > at hudson.Launcher$ProcStarter.start(Launcher.java:455)
> > at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2038)
> > ... 15 more
> >
> >
> >
> >
> > Enrico Olivelli
> >
> > BookKeeper committer
>
>
Re: OutOfMemory on Jenkins Slave H37
Posted by Chris Lambertus <cm...@apache.org>.
I rebooted the machine, but it didn’t come back up. Waiting on the colo to take a look at it.
> On Sep 24, 2019, at 7:36 AM, Enrico Olivelli <eo...@gmail.com> wrote:
>
> Hello,
> I see this OutOfMemoryError on H37, can you please take a look ?
>
> https://builds.apache.org/job/bookkeeper_precommit_replication_tests/1244/
>
> hudson.plugins.git.GitException: Could not init
> /home/jenkins/jenkins-slave/workspace/bookkeeper_precommit_replication_tests
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:813)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:605)
> at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
> at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
> at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> at hudson.remoting.Request$2.run(Request.java:369)
> at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to H37
> at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> at hudson.remoting.Channel.call(Channel.java:955)
> at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
> at sun.reflect.GeneratedMethodAccessor1084.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
> at com.sun.proxy.$Proxy135.execute(Unknown Source)
> at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152)
> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> at hudson.scm.SCM.checkout(SCM.java:504)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> at hudson.model.Run.execute(Run.java:1810)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> at hudson.model.ResourceController.execute(ResourceController.java:97)
> at hudson.model.Executor.run(Executor.java:429)
> Caused by: hudson.plugins.git.GitException: Error performing git command
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2051)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:811)
> ... 11 more
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:717)
> at hudson.Proc$LocalProc.<init>(Proc.java:269)
> at hudson.Proc$LocalProc.<init>(Proc.java:218)
> at hudson.Launcher$LocalLauncher.launch(Launcher.java:936)
> at hudson.Launcher$ProcStarter.start(Launcher.java:455)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2038)
> ... 15 more
>
>
>
>
> Enrico Olivelli
>
> BookKeeper committer