You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Busy Dev <bu...@gmail.com> on 2012/10/02 12:48:52 UTC
System VMs will not start
I've been trying to get my CS environment back up and have run into this
issue several times. The relevant log snippets are below. What am I
missing?
Thanks.
*libvirtd.log shows:*
2012-10-02 10:44:22.590+0000: 1777: warning : qemuDomainObjTaint:1297 :
Domain id=10 name='v-2-TEST' uuid=89c170ae-6bd5-38c6-bbdf-81f4174a05db is
tainted: high-privileges
2012-10-02 10:44:22.678+0000: 1777: error : virSecurityDACSetOwnership:125
: unable to set user and group to '0:0' on
'/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-7fd5438aea3e':
No such file or directory
2012-10-02 10:44:22.879+0000: 1777: error :
virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-7fd5438aea3e:
No such file or directory
*agent.log shows:*
2012-10-02 06:44:22,942 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) Failed to start domain: v-2-TEST: unable to
set user and group to '0:0' on
'/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-7fd5438aea3e':
No such file or directory
2012-10-02 06:44:22,943 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) Exception
org.libvirt.LibvirtException: unable to set user and group to '0:0' on
'/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-7fd5438aea3e':
No such file or directory
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Domain.processError(Unknown Source)
at org.libvirt.Domain.create(Unknown Source)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startDomain(LibvirtComputingResource.java:910)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2812)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1035)
at com.cloud.agent.Agent.processRequest(Agent.java:518)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-10-02 06:44:22,943 WARN [cloud.agent.Agent]
(agentRequest-Handler-1:null) Caught:
java.lang.NullPointerException
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupVMNetworks(LibvirtComputingResource.java:3921)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.handleVmStartFailure(LibvirtComputingResource.java:2708)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2833)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1035)
at com.cloud.agent.Agent.processRequest(Agent.java:518)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Re: System VMs will not start
Posted by Busy Dev <bu...@gmail.com>.
The management server log does not contain the string f64c8297 - grep
returns nothing.
[root@kvmhost ~]# for image in `ls
/mnt/b144de1c-1093-3cd1-96d1-4817dee90368`; do qemu-img info $image;
done
qemu-img: Could not open '712bf3a8-90b6-4216-a887-d88117d8d627'
qemu-img: Could not open '9ffa5e78-1ac3-47ac-a9f3-9c473b6792da'
qemu-img: Could not open 'd876191c-a171-4868-9736-aaf00c2e9d20'
qemu-img: Could not open 'KVMHA'
qemu-img: Could not open 's-1-TEST-patchdisk'
qemu-img: Could not open 's-3-TEST-patchdisk'
qemu-img: Could not open 's-4-TEST-patchdisk'
qemu-img: Could not open 'v-2-TEST-patchdisk'
On Tue, Oct 2, 2012 at 1:08 PM, Edison Su <Ed...@citrix.com> wrote:
> Can you search f64c8297-5017-4201-952a-7fd5438aea3e in your mgt server log?
> And run the following command on your kvm host:
> for image in `ls /mnt/ b144de1c-1093-3cd1-96d1-4817dee90368`;
> do
> qemu-img info $image
> done
>
> Will you find any disk has backing storage points to
> f64c8297-5017-4201-952a-7fd5438aea3e?
>
> > -----Original Message-----
> > From: Busy Dev [mailto:busydev6@gmail.com]
> > Sent: Tuesday, October 02, 2012 4:12 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: System VMs will not start
> >
> > The directory (f64c8297-5017-4201-952a-7fd5438aea3e) it tries to access
> > is
> > not available under /mnt/b144de1c-1093-3cd1-96d1-4817dee90368
> >
> > [root@kvmhost b144de1c-1093-3cd1-96d1-4817dee90368]# ls -la
> > /mnt/b144de1c-1093-3cd1-96d1-4817dee90368/
> > total 741812
> > drwxrwxrwx. 3 root root 4096 Oct 2 06:43 .
> > drwxr-xr-x. 4 root root 4096 Oct 2 07:08 ..
> > -rwxr--r--. 1 root root 262144 Oct 2 00:25
> > 712bf3a8-90b6-4216-a887-d88117d8d627
> > -rwxr--r--. 1 root root 725811200 Oct 2 00:24
> > 9ffa5e78-1ac3-47ac-a9f3-9c473b6792da
> > -rwxr--r--. 1 root root 29163520 Oct 2 2012
> > bf1b19ff-c2a8-4fee-a298-5053567afa2a
> > drwxr-xr-x. 2 root root 4096 Oct 2 00:24 KVMHA
> > -rwxr--r--. 1 root root 10485760 Oct 2 06:34 s-1-TEST-patchdisk
> > -rwxr--r--. 1 root root 10485760 Oct 2 06:44 s-3-TEST-patchdisk
> > -rwxr--r--. 1 root root 10485760 Oct 2 2012 v-2-TEST-patchdisk
> >
> > Where does libvirtd get this information from?
> >
> > Thanks
> >
> >
> >
> > On Tue, Oct 2, 2012 at 6:48 AM, Busy Dev <bu...@gmail.com> wrote:
> >
> > > I've been trying to get my CS environment back up and have run into
> > this
> > > issue several times. The relevant log snippets are below. What am I
> > > missing?
> > >
> > > Thanks.
> > >
> > >
> > > *libvirtd.log shows:*
> > >
> > > 2012-10-02 10:44:22.590+0000: 1777: warning :
> > qemuDomainObjTaint:1297 :
> > > Domain id=10 name='v-2-TEST' uuid=89c170ae-6bd5-38c6-bbdf-
> > 81f4174a05db is
> > > tainted: high-privileges
> > > 2012-10-02 10:44:22.678+0000: 1777: error :
> > virSecurityDACSetOwnership:125
> > > : unable to set user and group to '0:0' on
> > > '/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-
> > 7fd5438aea3e':
> > > No such file or directory
> > > 2012-10-02 10:44:22.879+0000: 1777: error :
> > > virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
> > > /mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-
> > 7fd5438aea3e:
> > > No such file or directory
> > >
> > >
> > > *agent.log shows:*
> > >
> > > 2012-10-02 06:44:22,942 WARN [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-1:null) Failed to start domain: v-2-TEST:
> > unable to
> > > set user and group to '0:0' on
> > > '/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-
> > 7fd5438aea3e':
> > > No such file or directory
> > > 2012-10-02 06:44:22,943 WARN [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-1:null) Exception
> > > org.libvirt.LibvirtException: unable to set user and group to '0:0'
> > on
> > > '/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-
> > 7fd5438aea3e':
> > > No such file or directory
> > > at org.libvirt.ErrorHandler.processError(Unknown Source)
> > > at org.libvirt.Connect.processError(Unknown Source)
> > > at org.libvirt.Domain.processError(Unknown Source)
> > > at org.libvirt.Domain.create(Unknown Source)
> > > at
> > >
> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startDomain(
> > LibvirtComputingResource.java:910)
> > > at
> > >
> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(Libv
> > irtComputingResource.java:2812)
> > > at
> > >
> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeReque
> > st(LibvirtComputingResource.java:1035)
> > > at com.cloud.agent.Agent.processRequest(Agent.java:518)
> > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831)
> > > at com.cloud.utils.nio.Task.run(Task.java:83)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
> > va:1110)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> > ava:603)
> > > at java.lang.Thread.run(Thread.java:679)
> > > 2012-10-02 06:44:22,943 WARN [cloud.agent.Agent]
> > > (agentRequest-Handler-1:null) Caught:
> > > java.lang.NullPointerException
> > > at
> > >
> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupVMNet
> > works(LibvirtComputingResource.java:3921)
> > > at
> > >
> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.handleVmStar
> > tFailure(LibvirtComputingResource.java:2708)
> > > at
> > >
> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(Libv
> > irtComputingResource.java:2833)
> > > at
> > >
> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeReque
> > st(LibvirtComputingResource.java:1035)
> > > at com.cloud.agent.Agent.processRequest(Agent.java:518)
> > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831)
> > > at com.cloud.utils.nio.Task.run(Task.java:83)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
> > va:1110)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> > ava:603)
> > > at java.lang.Thread.run(Thread.java:679)
> > >
> > >
>
RE: System VMs will not start
Posted by Edison Su <Ed...@citrix.com>.
Can you search f64c8297-5017-4201-952a-7fd5438aea3e in your mgt server log?
And run the following command on your kvm host:
for image in `ls /mnt/ b144de1c-1093-3cd1-96d1-4817dee90368`;
do
qemu-img info $image
done
Will you find any disk has backing storage points to f64c8297-5017-4201-952a-7fd5438aea3e?
> -----Original Message-----
> From: Busy Dev [mailto:busydev6@gmail.com]
> Sent: Tuesday, October 02, 2012 4:12 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: System VMs will not start
>
> The directory (f64c8297-5017-4201-952a-7fd5438aea3e) it tries to access
> is
> not available under /mnt/b144de1c-1093-3cd1-96d1-4817dee90368
>
> [root@kvmhost b144de1c-1093-3cd1-96d1-4817dee90368]# ls -la
> /mnt/b144de1c-1093-3cd1-96d1-4817dee90368/
> total 741812
> drwxrwxrwx. 3 root root 4096 Oct 2 06:43 .
> drwxr-xr-x. 4 root root 4096 Oct 2 07:08 ..
> -rwxr--r--. 1 root root 262144 Oct 2 00:25
> 712bf3a8-90b6-4216-a887-d88117d8d627
> -rwxr--r--. 1 root root 725811200 Oct 2 00:24
> 9ffa5e78-1ac3-47ac-a9f3-9c473b6792da
> -rwxr--r--. 1 root root 29163520 Oct 2 2012
> bf1b19ff-c2a8-4fee-a298-5053567afa2a
> drwxr-xr-x. 2 root root 4096 Oct 2 00:24 KVMHA
> -rwxr--r--. 1 root root 10485760 Oct 2 06:34 s-1-TEST-patchdisk
> -rwxr--r--. 1 root root 10485760 Oct 2 06:44 s-3-TEST-patchdisk
> -rwxr--r--. 1 root root 10485760 Oct 2 2012 v-2-TEST-patchdisk
>
> Where does libvirtd get this information from?
>
> Thanks
>
>
>
> On Tue, Oct 2, 2012 at 6:48 AM, Busy Dev <bu...@gmail.com> wrote:
>
> > I've been trying to get my CS environment back up and have run into
> this
> > issue several times. The relevant log snippets are below. What am I
> > missing?
> >
> > Thanks.
> >
> >
> > *libvirtd.log shows:*
> >
> > 2012-10-02 10:44:22.590+0000: 1777: warning :
> qemuDomainObjTaint:1297 :
> > Domain id=10 name='v-2-TEST' uuid=89c170ae-6bd5-38c6-bbdf-
> 81f4174a05db is
> > tainted: high-privileges
> > 2012-10-02 10:44:22.678+0000: 1777: error :
> virSecurityDACSetOwnership:125
> > : unable to set user and group to '0:0' on
> > '/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-
> 7fd5438aea3e':
> > No such file or directory
> > 2012-10-02 10:44:22.879+0000: 1777: error :
> > virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
> > /mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-
> 7fd5438aea3e:
> > No such file or directory
> >
> >
> > *agent.log shows:*
> >
> > 2012-10-02 06:44:22,942 WARN [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-1:null) Failed to start domain: v-2-TEST:
> unable to
> > set user and group to '0:0' on
> > '/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-
> 7fd5438aea3e':
> > No such file or directory
> > 2012-10-02 06:44:22,943 WARN [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-1:null) Exception
> > org.libvirt.LibvirtException: unable to set user and group to '0:0'
> on
> > '/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-
> 7fd5438aea3e':
> > No such file or directory
> > at org.libvirt.ErrorHandler.processError(Unknown Source)
> > at org.libvirt.Connect.processError(Unknown Source)
> > at org.libvirt.Domain.processError(Unknown Source)
> > at org.libvirt.Domain.create(Unknown Source)
> > at
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startDomain(
> LibvirtComputingResource.java:910)
> > at
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(Libv
> irtComputingResource.java:2812)
> > at
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeReque
> st(LibvirtComputingResource.java:1035)
> > at com.cloud.agent.Agent.processRequest(Agent.java:518)
> > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831)
> > at com.cloud.utils.nio.Task.run(Task.java:83)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
> va:1110)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> ava:603)
> > at java.lang.Thread.run(Thread.java:679)
> > 2012-10-02 06:44:22,943 WARN [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Caught:
> > java.lang.NullPointerException
> > at
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupVMNet
> works(LibvirtComputingResource.java:3921)
> > at
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.handleVmStar
> tFailure(LibvirtComputingResource.java:2708)
> > at
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(Libv
> irtComputingResource.java:2833)
> > at
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeReque
> st(LibvirtComputingResource.java:1035)
> > at com.cloud.agent.Agent.processRequest(Agent.java:518)
> > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831)
> > at com.cloud.utils.nio.Task.run(Task.java:83)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
> va:1110)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> ava:603)
> > at java.lang.Thread.run(Thread.java:679)
> >
> >
Re: System VMs will not start
Posted by Busy Dev <bu...@gmail.com>.
The directory (f64c8297-5017-4201-952a-7fd5438aea3e) it tries to access is
not available under /mnt/b144de1c-1093-3cd1-96d1-4817dee90368
[root@kvmhost b144de1c-1093-3cd1-96d1-4817dee90368]# ls -la
/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/
total 741812
drwxrwxrwx. 3 root root 4096 Oct 2 06:43 .
drwxr-xr-x. 4 root root 4096 Oct 2 07:08 ..
-rwxr--r--. 1 root root 262144 Oct 2 00:25
712bf3a8-90b6-4216-a887-d88117d8d627
-rwxr--r--. 1 root root 725811200 Oct 2 00:24
9ffa5e78-1ac3-47ac-a9f3-9c473b6792da
-rwxr--r--. 1 root root 29163520 Oct 2 2012
bf1b19ff-c2a8-4fee-a298-5053567afa2a
drwxr-xr-x. 2 root root 4096 Oct 2 00:24 KVMHA
-rwxr--r--. 1 root root 10485760 Oct 2 06:34 s-1-TEST-patchdisk
-rwxr--r--. 1 root root 10485760 Oct 2 06:44 s-3-TEST-patchdisk
-rwxr--r--. 1 root root 10485760 Oct 2 2012 v-2-TEST-patchdisk
Where does libvirtd get this information from?
Thanks
On Tue, Oct 2, 2012 at 6:48 AM, Busy Dev <bu...@gmail.com> wrote:
> I've been trying to get my CS environment back up and have run into this
> issue several times. The relevant log snippets are below. What am I
> missing?
>
> Thanks.
>
>
> *libvirtd.log shows:*
>
> 2012-10-02 10:44:22.590+0000: 1777: warning : qemuDomainObjTaint:1297 :
> Domain id=10 name='v-2-TEST' uuid=89c170ae-6bd5-38c6-bbdf-81f4174a05db is
> tainted: high-privileges
> 2012-10-02 10:44:22.678+0000: 1777: error : virSecurityDACSetOwnership:125
> : unable to set user and group to '0:0' on
> '/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-7fd5438aea3e':
> No such file or directory
> 2012-10-02 10:44:22.879+0000: 1777: error :
> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
> /mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-7fd5438aea3e:
> No such file or directory
>
>
> *agent.log shows:*
>
> 2012-10-02 06:44:22,942 WARN [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:null) Failed to start domain: v-2-TEST: unable to
> set user and group to '0:0' on
> '/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-7fd5438aea3e':
> No such file or directory
> 2012-10-02 06:44:22,943 WARN [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:null) Exception
> org.libvirt.LibvirtException: unable to set user and group to '0:0' on
> '/mnt/b144de1c-1093-3cd1-96d1-4817dee90368/f64c8297-5017-4201-952a-7fd5438aea3e':
> No such file or directory
> at org.libvirt.ErrorHandler.processError(Unknown Source)
> at org.libvirt.Connect.processError(Unknown Source)
> at org.libvirt.Domain.processError(Unknown Source)
> at org.libvirt.Domain.create(Unknown Source)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startDomain(LibvirtComputingResource.java:910)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2812)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1035)
> at com.cloud.agent.Agent.processRequest(Agent.java:518)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831)
> at com.cloud.utils.nio.Task.run(Task.java:83)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2012-10-02 06:44:22,943 WARN [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Caught:
> java.lang.NullPointerException
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupVMNetworks(LibvirtComputingResource.java:3921)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.handleVmStartFailure(LibvirtComputingResource.java:2708)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2833)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1035)
> at com.cloud.agent.Agent.processRequest(Agent.java:518)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831)
> at com.cloud.utils.nio.Task.run(Task.java:83)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>