You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Michael Kolomiets <mi...@gmail.com> on 2014/11/27 20:39:56 UTC
Agent SIGSEGV
Hi
I have some problems with CS 4.4.1 - agent was killed with SIGSEGV.
Please, help me to detect a source of problem.
System Ubuntu Server 14.04 LTS, hypervisor KVM, Ceph RBD as primary
storage, secondary storage - Ceph radosgw (S3).
There is records from apport.log:
root@lw01p01-node03:~# cat /var/log/apport.log
ERROR: apport (pid 43770) Thu Nov 27 09:21:49 2014: called for pid 30814,
signal 11, core limit 0
ERROR: apport (pid 43770) Thu Nov 27 09:21:49 2014: executable:
/usr/bin/jsvc (command line "jsvc.exec -cp
/usr/share/java/commons-daemon.jar:/usr/share/cloudstack-agent/lib/activation-1.1.jar:/usr/share/cloudstack-agent/lib/antisamy-1.4.3.jar:/usr/share/cloudstack-agent/lib/aopalliance-1.0.jar:/usr/share/cloudstack-agent/lib/apache-log4j-extras-1.1.jar:/usr/share/cloudstack-agent/lib/aspectjweaver-1.7.0.jar:/usr/share/cloudstack-agent/lib/aws-java-sdk-1.3.22.jar:/usr/share/cloudstack-agent/lib/batik-css-1.7.jar:/usr/share/cloudstack-agent/lib/batik-ext-1.7.jar:/usr/share/cloudstack-agent/lib/batik-util-1.7.jar:/usr/share/cloudstack-agent/lib/bcprov-jdk16-1.46.jar:/usr/share/cloudstack-agent/lib/bsh-core-2.0b4.jar:/usr/share/cloudstack-agent/lib/cglib-nodep-2.2.2.jar:/usr/share/cloudstack-agent/lib/cloud-agent-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-core-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-components-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-schema-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-cluster-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-config-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-db-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-events-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-ipc-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-jobs-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-managed-context-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-rest-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-security-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-plugin-hypervisor-kvm-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-plugin-network-ovs-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-server-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloudstack-agent.jar:/usr/share/cloudstack-agent/lib/cloud-utils-4.4.1.jar:/usr/share/cloudstack-agent/lib/commons-beanutils-core-1.7.0.jar:/usr/share/cloudstack-agent/lib/commons-codec-1.6.jar:/usr/share/cloudstack-agent/lib/commons-collections-3.2.jar:/usr/share/cloudstack-agent/lib/commons-configuration-1.8.jar:/usr/share/cloudstack-agent/lib/commons-daemon-1.0.10.jar:/usr/share/cloudstack-agent/lib/commons-dbcp-1.4.jar:/usr/share/cloudstack-agent/lib/commons-fileupload-1.2.jar:/usr/share/cloudstack-agent/lib/commons-httpclient-3.1.jar:/usr/share/cloudstack-agent/lib/commons-io-1.4.jar:/usr/share/cloudstack-agent/lib/commons-lang-2.6.jar:/usr/share/cloudstack-agent/lib/commons-logging-1.1.1.jar:/usr/share/cloudstack-agent/lib/commons-net-3.3.jar:/usr/share/cloudstack-agent/lib/commons-pool-1.6.jar:/usr/share/cloudstack-agent/lib/cxf-bundle-jaxrs-2.7.0.jar:/usr/share/cloudstack-agent/lib/dom4j-1.6.1.jar:/usr/share/cloudstack-agent/lib/ehcache-core-2.6.6.jar:/usr/share/cloudstack-agent/lib/ejb-api-3.0.jar:/usr/share/cloudstack-agent/lib/esapi-2.0.1.jar:/usr/share/cloudstack-agent/lib/geronimo-javamail_1.4_spec-1.7.1.jar:/usr/share/cloudstack-agent/lib/geronimo-servlet_3.0_spec-1.0.jar:/usr/share/cloudstack-agent/lib/gson-1.7.2.jar:/usr/share/cloudstack-agent/lib/guava-14.0-rc1.jar:/usr/share/cloudstack-agent/lib/httpclient-4.2.1.jar:/usr/share/cloudstack-agent/lib/httpcore-4.2.1.jar:/usr/share/cloudstack-agent/lib/jackson-annotations-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-core-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-core-asl-1.8.9.jar:/usr/share/cloudstack-agent/lib/jackson-databind-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-jaxrs-json-provider-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-mapper-asl-1.8.9.jar:/usr/share/cloudstack-agent/lib/jackson-module-jaxb-annotations-2.1.1.jar:/usr/share/cloudstack-agent/lib/jasypt-1.9.0.jar:/usr/share/cloudstack-agent/lib/java-ipv6-0.10.jar:/usr/share/cloudstack-agent/lib/javassist-3.12.1.GA.jar:/usr/share/cloudstack-agent/lib/javassist-3.18.1-GA.jar:/usr/share/cloudstack-agent/lib/javax.inject-1.jar:/usr/share/cloudstack-agent/lib/javax.persistence-2.0.0.jar:/usr/share/cloudstack-agent/lib/javax.ws.rs-api-2.0-m10.jar:/usr/share/cloudstack")
ERROR: apport (pid 43770) Thu Nov 27 09:21:49 2014: is_closing_session():
no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 43770) Thu Nov 27 09:21:49 2014: apport: report
/var/crash/_usr_bin_jsvc.0.crash already exists and unseen, doing nothing
to avoid disk usage DoS
Crash reports are to big to post in message, next is top lines of one.
root@lw01p01-node03:~# head -100 /var/crash/_usr_bin_jsvc.0.crash
ProblemType: Crash
Architecture: amd64
Date: Wed Nov 19 14:44:28 2014
DistroRelease: Ubuntu 14.04
ExecutablePath: /usr/bin/jsvc
ExecutableTimestamp: 1397734309
ProcCmdline: jsvc.exec -cp
/usr/share/java/commons-daemon.jar:/usr/share/cloudstack-agent/lib/activation-1.1.jar:/usr/share/cloudstack-agent/lib/antisamy-1.4.3.jar:/usr/share/cloudstack-agent/lib/aopalliance-1.0.jar:/usr/share/cloudstack-agent/lib/apache-log4j-extras-1.1.jar:/usr/share/cloudstack-agent/lib/aspectjweaver-1.7.0.jar:/usr/share/cloudstack-agent/lib/aws-java-sdk-1.3.22.jar:/usr/share/cloudstack-agent/lib/batik-css-1.7.jar:/usr/share/cloudstack-agent/lib/batik-ext-1.7.jar:/usr/share/cloudstack-agent/lib/batik-util-1.7.jar:/usr/share/cloudstack-agent/lib/bcprov-jdk16-1.46.jar:/usr/share/cloudstack-agent/lib/bsh-core-2.0b4.jar:/usr/share/cloudstack-agent/lib/cglib-nodep-2.2.2.jar:/usr/share/cloudstack-agent/lib/cloud-agent-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-core-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-components-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-schema-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-cluster-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-config-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-db-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-events-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-ipc-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-jobs-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-managed-context-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-rest-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-security-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-plugin-hypervisor-kvm-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-plugin-network-ovs-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-server-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloudstack-agent.jar:/usr/share/cloudstack-agent/lib/cloud-utils-4.4.1.jar:/usr/share/cloudstack-agent/lib/commons-beanutils-core-1.7.0.jar:/usr/share/cloudstack-agent/lib/commons-codec-1.6.jar:/usr/share/cloudstack-agent/lib/commons-collections-3.2.jar:/usr/share/cloudstack-agent/lib/commons-configuration-1.8.jar:/usr/share/cloudstack-agent/lib/commons-daemon-1.0.10.jar:/usr/share/cloudstack-agent/lib/commons-dbcp-1.4.jar:/usr/share/cloudstack-agent/lib/commons-fileupload-1.2.jar:/usr/share/cloudstack-agent/lib/commons-httpclient-3.1.jar:/usr/share/cloudstack-agent/lib/commons-io-1.4.jar:/usr/share/cloudstack-agent/lib/commons-lang-2.6.jar:/usr/share/cloudstack-agent/lib/commons-logging-1.1.1.jar:/usr/share/cloudstack-agent/lib/commons-net-3.3.jar:/usr/share/cloudstack-agent/lib/commons-pool-1.6.jar:/usr/share/cloudstack-agent/lib/cxf-bundle-jaxrs-2.7.0.jar:/usr/share/cloudstack-agent/lib/dom4j-1.6.1.jar:/usr/share/cloudstack-agent/lib/ehcache-core-2.6.6.jar:/usr/share/cloudstack-agent/lib/ejb-api-3.0.jar:/usr/share/cloudstack-agent/lib/esapi-2.0.1.jar:/usr/share/cloudstack-agent/lib/geronimo-javamail_1.4_spec-1.7.1.jar:/usr/share/cloudstack-agent/lib/geronimo-servlet_3.0_spec-1.0.jar:/usr/share/cloudstack-agent/lib/gson-1.7.2.jar:/usr/share/cloudstack-agent/lib/guava-14.0-rc1.jar:/usr/share/cloudstack-agent/lib/httpclient-4.2.1.jar:/usr/share/cloudstack-agent/lib/httpcore-4.2.1.jar:/usr/share/cloudstack-agent/lib/jackson-annotations-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-core-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-core-asl-1.8.9.jar:/usr/share/cloudstack-agent/lib/jackson-databind-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-jaxrs-json-provider-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-mapper-asl-1.8.9.jar:/usr/share/cloudstack-agent/lib/jackson-module-jaxb-annotations-2.1.1.jar:/usr/share/cloudstack-agent/lib/jasypt-1.9.0.jar:/usr/share/cloudstack-agent/lib/java-ipv6-0.10.jar:/usr/share/cloudstack-agent/lib/javassist-3.12.1.GA.jar:/usr/share/cloudstack-agent/lib/javassist-3.18.1-GA.jar:/usr/share/cloudstack-agent/lib/javax.inject-1.jar:/usr/share/cloudstack-agent/lib/javax.persistence-2.0.0.jar:/usr/share/cloudstack-agent/lib/javax.ws.rs-api-2.0-m10.jar:/usr/share/cloudstack
ProcCwd: /
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=en_US.UTF-8
LD_LIBRARY_PATH=<set>
ProcMaps:
00400000-0040b000 r-xp 00000000 fc:01 1068396
/usr/bin/jsvc
0060a000-0060b000 r--p 0000a000 fc:01 1068396
/usr/bin/jsvc
0060b000-0060c000 rw-p 0000b000 fc:01 1068396
/usr/bin/jsvc
0154c000-0156d000 rw-p 00000000 00:00 0
[heap]
0156d000-01e38000 rw-p 00000000 00:00 0
[heap]
407a00000-408f00000 rw-p 00000000 00:00 0
...
and also:
...
7fffa6ffe000-7fffa7000000 r-xp 00000000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
ProcStatus:
Name: jsvc
State: D (disk sleep)
Tgid: 27041
Ngid: 27041
Pid: 27041
PPid: 27038
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 256
Groups: 0
VmPeak: 22568424 kB
VmSize: 22444004 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 376728 kB
VmRSS: 249872 kB
VmData: 22250264 kB
VmStk: 1012 kB
VmExe: 44 kB
VmLib: 37864 kB
VmPTE: 1732 kB
VmSwap: 30216 kB
Threads: 61
SigQ: 4/514887
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000004
SigIgn: 0000000000001000
SigCgt: 20000001810048cf
CapInh: 0000000000000000
CapPrm: 0000001fffffffff
CapEff: 0000001fffffffff
CapBnd: 0000001fffffffff
Seccomp: 0
Cpus_allowed: ffff,ffffffff
Cpus_allowed_list: 0-47
Mems_allowed: 00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 270184
nonvoluntary_ctxt_switches: 454
Signal: 11
Uname: Linux 3.14.1-031401-generic x86_64
UserGroups:
_LogindSession: /user/1000.user/3.session
CoreDump: base64
...
--
Michael Kolomiets