You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Satish Chhatpar 02 <Ch...@cpwplc.com> on 2017/01/26 19:28:51 UTC

Apache Tomcat/7.0.39 crashed with fatal error

Hi,


we are using Apache Tomcat/7.0.39 for our java application. There are 4 tomcat instances using same tomcat version and java version. yesterday all 4 tomcats crashed with below error in hs_err_pid log file.

This log file was created for all 4 tomcats.

Its very peculiar behaviour that all 4 crashed around same time.


Any information can help us to mitigate this incident.




Apache Tomcat/7.0.39



java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)


OS used


Red Hat Enterprise Linux Server release 6.3 (Santiago)



#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fed24ecfe9e, pid=21352, tid=140656275650304
#
# JRE version: 7.0_21-b11
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J  org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List;
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007fecf832f000):  JavaThread "AD Thread-Metric Reporter0" daemon [_thread_in_Java, id=21367, stack(0x00007fed17487000,0x00007fed17588000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000003f09daccec

Registers:
RAX=0x00000007e14683e0, RBX=0x0000000000000000, RCX=0x0000003f09dacce0, RDX=0x0000000000000005
RSP=0x00007fed17585ee0, RBP=0x00000000eda7c0dd, RSI=0x00000007e13b59a0, RDI=0x00000007e13b599c
R8 =0x0000000000000004, R9 =0x0000000000000003, R10=0x0000000000000000, R11=0x0000000000000001
R12=0x0000000000000000, R13=0x0000000660a90c70, R14=0x0000000000000000, R15=0x00007fecf832f000
RIP=0x00007fed24ecfe9e, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fed17585ee0)
0x00007fed17585ee0:   cc15218e00000001 00007fed0000000a
0x00007fed17585ef0:   00000007e11fec98 0000000660a90c58
0x00007fed17585f00:   0000000660a90b90 0000000000000005
0x00007fed17585f10:   00000007e11fecd0 0000000000000000
0x00007fed17585f20:   00007fed17585f50 000000072057de50
0x00007fed17585f30:   00007fed24abd218 0000000660a90b70
0x00007fed17585f40:   000000072057de50 00007fed24abe508
0x00007fed17585f50:   0000000660a90b90 0000000660a904a8
0x00007fed17585f60:   00000007220723e0 0000000660a90c58
0x00007fed17585f70:   0000000660a90b90 0000000660a90bb0
0x00007fed17585f80:   00000007220723e0 0000000660a90ca8
0x00007fed17585f90:   0000000660a90c58 0000000660a90b70
0x00007fed17585fa0:   0000000400000005 0000000600000000
0x00007fed17585fb0:   0000000660a90bb0 00000007220cc188
0x00007fed17585fc0:   ffffffff24188870 000000072206f478
0x00007fed17585fd0:   0000000660a90b90 00000006602f73e8
0x00007fed17585fe0:   0000000000000009 00000007220723e0
0x00007fed17585ff0:   0000000000000006 00007fed24188870
0x00007fed17586000:   0000000660a904a8 00000007220723e0
0x00007fed17586010:   00000007220bea48 00007fed00000009
0x00007fed17586020:   00000007e0279348 00000007eb43afd0
0x00007fed17586030:   00007fed175860a0 00007fed23173806
0x00007fed17586040:   00007fed23173806 00000007220723e0
0x00007fed17586050:   0000000660a904a8 00000007220d20f0
0x00007fed17586060:   00007fed17586060 00000007e0a8092e
0x00007fed17586070:   00007fed17586108 00000007e0a80db8
0x00007fed17586080:   00000007eb367008 00000007e0a80ab0
0x00007fed17586090:   00007fed17586048 00007fed175860e8
0x00007fed175860a0:   00007fed17586150 00007fed231739e1
0x00007fed175860b0:   0000000000000000 0000000000000000
0x00007fed175860c0:   000000072206dfe8 0000000000000000
0x00007fed175860d0:   0000000660a90878 00000006602f7268

Instructions: (pc=0x00007fed24ecfe9e)
0x00007fed24ecfe7e:   85 04 00 00 00 48 89 46 28 41 0f 45 d9 45 8b cb
0x00007fed24ecfe8e:   41 83 c1 02 49 63 cb 41 8b 6c 8d 14 49 8d 0c fc
0x00007fed24ecfe9e:   8b 79 0c 41 3b fa 44 0f 4c d7 85 ed 0f 84 59 08
0x00007fed24ecfeae:   00 00 41 8b 7c ec 08 81 ff 34 6b 27 fc 0f 85 b9

Register to memory mapping:

RAX=0x00000007e14683e0 is an oop
{instance class}
 - klass: {other class}
RBX=0x0000000000000000 is an unknown value
RCX=0x0000003f09dacce0 is an unknown value
RDX=0x0000000000000005 is an unknown value
RSP=0x00007fed17585ee0 is pointing into the stack for thread: 0x00007fecf832f000
RBP=0x00000000eda7c0dd is an unknown value
RSI=0x00000007e13b59a0 is an oop
{instance class}
 - klass: {other class}
RDI=0x00000007e13b599c is an oop
[[I
 - klass: 'java/lang/Object'[]
 - length: 5
R8 =0x0000000000000004 is an unknown value
R9 =0x0000000000000003 is an unknown value
R10=0x0000000000000000 is an unknown value
R11=0x0000000000000001 is an unknown value
R12=0x0000000000000000 is an unknown value
R13=
[error occurred during error reporting (printing register info), id 0xb]

Stack: [0x00007fed17487000,0x00007fed17588000],  sp=0x00007fed17585ee0,  free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J  org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List;


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00007fecec249800 JavaThread "AD Thread Pool-Global26" daemon [_thread_blocked, id=20797, stack(0x00007feccdcdd000,0x00007feccddde000)]
  0x00007fecec0cc000 JavaThread "AD Thread Pool-Global23" daemon [_thread_blocked, id=10466, stack(0x00007feccfdfe000,0x00007feccfeff000)]
  0x00007fed0c037800 JavaThread "AD Thread Pool-CSS1" daemon [_thread_blocked, id=25552, stack(0x00007feccf2f3000,0x00007feccf3f4000)]
  0x0000000001dce800 JavaThread "AD Thread Pool-CGG0" daemon [_thread_blocked, id=24644, stack(0x00007fecd5f68000,0x00007fecd6069000)]
  0x0000000001dd1000 JavaThread "AD Thread Pool-CSS0" daemon [_thread_blocked, id=24643, stack(0x00007fecd636c000,0x00007fecd646d000)]
  0x00007fed10525000 JavaThread "task-scheduler-10" [_thread_blocked, id=23743, stack(0x00007fecd5c65000,0x00007fecd5d66000)]
  0x00007fecf01ca000 JavaThread "task-scheduler-9" [_thread_blocked, id=23658, stack(0x00007fecd5059000,0x00007fecd515a000)]
  0x00007fed0858f800 JavaThread "task-scheduler-8" [_thread_blocked, id=23631, stack(0x00007fecd6c75000,0x00007fecd6d76000)]
  0x00007fed2c2a6800 JavaThread "task-scheduler-7" [_thread_blocked, id=23613, stack(0x00007feccd4d5000,0x00007feccd5d6000)]
  0x00007fed0c00f800 JavaThread "task-scheduler-6" [_thread_blocked, id=23519, stack(0x00007fecd5962000,0x00007fecd5a63000)]
  0x00007fecf0231800 JavaThread "task-scheduler-5" [_thread_blocked, id=23191, stack(0x00007fecd414a000,0x00007fecd424b000)]
  0x00007fed2cc40000 JavaThread "task-scheduler-4" [_thread_blocked, id=23172, stack(0x00007fecd4e57000,0x00007fecd4f58000)]
  0x00007fecf004d800 JavaThread "task-scheduler-3" [_thread_blocked, id=23155, stack(0x00007fed1631e000,0x00007fed1641f000)]
  0x0000000001dd2800 JavaThread "AD Thread-Top Backends Processor0" daemon [_thread_blocked, id=23138, stack(0x00007feccc745000,0x00007feccc846000)]
  0x00007fecfc028000 JavaThread "executor-10" [_thread_blocked, id=23136, stack(0x00007feccc846000,0x00007feccc947000)]
  0x00007fecfc023000 JavaThread "executor-9" [_thread_blocked, id=23135, stack(0x00007feccc947000,0x00007feccca48000)]
  0x00007fed2cb72000 JavaThread "task-scheduler-10" [_thread_blocked, id=23134, stack(0x00007fecccdce000,0x00007fecccecf000)]
  0x00007fed2cb66000 JavaThread "executor-8" [_thread_blocked, id=23133, stack(0x00007fecccecf000,0x00007fecccfd0000)]
  0x00007fed083d8800 JavaThread "http-nio-8804-exec-97" daemon [_thread_blocked, id=23131, stack(0x00007feccd0d1000,0x00007feccd1d2000)]
  0x00007fed08243800 JavaThread "http-nio-8804-exec-95" daemon [_thread_blocked, id=23129, stack(0x00007feccd2d3000,0x00007feccd3d4000)]
  0x00007fed0a091800 JavaThread "http-nio-8804-exec-92" daemon [_thread_blocked, id=23126, stack(0x00007feccd5d6000,0x00007feccd6d7000)]
  0x00007fed08750000 JavaThread "http-nio-8804-exec-91" daemon [_thread_blocked, id=23125, stack(0x00007feccd6d7000,0x00007feccd7d8000)]
  0x00007fed0a172800 JavaThread "http-nio-8804-exec-90" daemon [_thread_blocked, id=23124, stack(0x00007feccd7d8000,0x00007feccd8d9000)]
  0x00007fed0994b800 JavaThread "http-nio-8804-exec-87" daemon [_thread_blocked, id=23121, stack(0x00007feccdadb000,0x00007feccdbdc000)]
  0x00007fed0886c800 JavaThread "http-nio-8804-exec-86" daemon [_thread_blocked, id=23120, stack(0x00007feccdbdc000,0x00007feccdcdd000)]
  0x00007fed08cff800 JavaThread "http-nio-8804-exec-84" daemon [_thread_blocked, id=23118, stack(0x00007feccddde000,0x00007feccdedf000)]
  0x00007fed08d35800 JavaThread "http-nio-8804-exec-82" daemon [_thread_in_Java, id=23116, stack(0x00007feccdfe0000,0x00007fecce0e1000)]
  0x00007fed080d2000 JavaThread "http-nio-8804-exec-81" daemon [_thread_blocked, id=23115, stack(0x00007fecce0e1000,0x00007fecce1e2000)]
  0x00007fed089ab800 JavaThread "http-nio-8804-exec-79" daemon [_thread_blocked, id=23113, stack(0x00007fecce2e3000,0x00007fecce3e4000)]
  0x00007fed09728000 JavaThread "http-nio-8804-exec-78" daemon [_thread_blocked, id=23112, stack(0x00007fecce3e4000,0x00007fecce4e5000)]
  0x00007fed098fd000 JavaThread "http-nio-8804-exec-77" daemon [_thread_blocked, id=23111, stack(0x00007fecce4e5000,0x00007fecce5e6000)]
  0x00007fed097e3800 JavaThread "http-nio-8804-exec-76" daemon [_thread_blocked, id=23110, stack(0x00007fecce5e6000,0x00007fecce6e7000)]
  0x00007fed097e2000 JavaThread "http-nio-8804-exec-75" daemon [_thread_blocked, id=23109, stack(0x00007fecce6e7000,0x00007fecce7e8000)]
  0x00007fed0901e000 JavaThread "http-nio-8804-exec-73" daemon [_thread_blocked, id=23107, stack(0x00007fecce8e9000,0x00007fecce9ea000)]
  0x00007fed08c88000 JavaThread "http-nio-8804-exec-71" daemon [_thread_blocked, id=23105, stack(0x00007fecceaeb000,0x00007feccebec000)]
  0x00007fed082dc800 JavaThread "http-nio-8804-exec-70" daemon [_thread_in_native, id=23104, stack(0x00007feccebec000,0x00007fecceced000)]
  0x00007fed093e1000 JavaThread "http-nio-8804-exec-69" daemon [_thread_blocked, id=23103, stack(0x00007fecceced000,0x00007feccedee000)]
  0x00007fed0858e000 JavaThread "http-nio-8804-exec-67" daemon [_thread_blocked, id=23101, stack(0x00007fecceeef000,0x00007fecceff0000)]
  0x00007fed08105800 JavaThread "http-nio-8804-exec-66" daemon [_thread_blocked, id=23100, stack(0x00007fecceff0000,0x00007feccf0f1000)]
  0x00007fed082e9800 JavaThread "http-nio-8804-exec-65" daemon [_thread_blocked, id=23099, stack(0x00007feccf0f1000,0x00007feccf1f2000)]
  0x00007fed09023000 JavaThread "http-nio-8804-exec-62" daemon [_thread_blocked, id=23096, stack(0x00007feccf3f4000,0x00007feccf4f5000)]
  0x00007fed09022000 JavaThread "http-nio-8804-exec-61" daemon [_thread_blocked, id=23095, stack(0x00007feccf4f5000,0x00007feccf5f6000)]
  0x00007fed09998000 JavaThread "http-nio-8804-exec-57" daemon [_thread_in_vm, id=23091, stack(0x00007feccf8f9000,0x00007feccf9fa000)]
  0x00007fed080ff000 JavaThread "http-nio-8804-exec-55" daemon [_thread_blocked, id=23089, stack(0x00007feccfafb000,0x00007feccfbfc000)]
  0x00007fed09986800 JavaThread "http-nio-8804-exec-51" daemon [_thread_blocked, id=23085, stack(0x00007feccfeff000,0x00007fecd0000000)]
  0x00007fed0c07d800 JavaThread "http-nio-8804-exec-49" daemon [_thread_blocked, id=23082, stack(0x00007fecd424b000,0x00007fecd434c000)]
  0x00007fed0c079000 JavaThread "http-nio-8804-exec-47" daemon [_thread_blocked, id=23080, stack(0x00007fecd444d000,0x00007fecd454e000)]
  0x00007fed0c077000 JavaThread "http-nio-8804-exec-46" daemon [_thread_blocked, id=23079, stack(0x00007fecd454e000,0x00007fecd464f000)]
  0x00007fed0c075000 JavaThread "http-nio-8804-exec-45" daemon [_thread_in_native, id=23078, stack(0x00007fecd464f000,0x00007fecd4750000)]
  0x00007fed0c06c800 JavaThread "http-nio-8804-exec-41" daemon [_thread_blocked, id=23074, stack(0x00007fecd4a53000,0x00007fecd4b54000)]
  0x00007fed0c06a800 JavaThread "http-nio-8804-exec-40" daemon [_thread_blocked, id=23073, stack(0x00007fecd4b54000,0x00007fecd4c55000)]
  0x00007fed0c062000 JavaThread "http-nio-8804-exec-36" daemon [_thread_blocked, id=23069, stack(0x00007fecd4f58000,0x00007fecd5059000)]
  0x00007fed0c05d800 JavaThread "http-nio-8804-exec-34" daemon [_thread_blocked, id=23067, stack(0x00007fecd515a000,0x00007fecd525b000)]
  0x00007fed0c059800 JavaThread "http-nio-8804-exec-32" daemon [_thread_blocked, id=23065, stack(0x00007fecd535c000,0x00007fecd545d000)]
  0x00007fed0c057800 JavaThread "http-nio-8804-exec-31" daemon [_thread_blocked, id=23064, stack(0x00007fecd545d000,0x00007fecd555e000)]
  0x00007fed0c053000 JavaThread "http-nio-8804-exec-29" daemon [_thread_blocked, id=23062, stack(0x00007fecd565f000,0x00007fecd5760000)]
  0x00007fed0c051000 JavaThread "http-nio-8804-exec-28" daemon [_thread_blocked, id=23061, stack(0x00007fecd5760000,0x00007fecd5861000)]
  0x00007fed0c04a800 JavaThread "http-nio-8804-exec-25" daemon [_thread_blocked, id=23058, stack(0x00007fecd5a63000,0x00007fecd5b64000)]
  0x00007fed0c044800 JavaThread "http-nio-8804-exec-22" daemon [_thread_blocked, id=23055, stack(0x00007fecd5d66000,0x00007fecd5e67000)]
  0x00007fed0c042000 JavaThread "http-nio-8804-exec-21" daemon [_thread_blocked, id=23054, stack(0x00007fecd5e67000,0x00007fecd5f68000)]
  0x00007fed0c03e000 JavaThread "http-nio-8804-exec-19" daemon [_thread_blocked, id=23052, stack(0x00007fecd6069000,0x00007fecd616a000)]
  0x00007fed0c03b800 JavaThread "http-nio-8804-exec-18" daemon [_thread_in_Java, id=23051, stack(0x00007fecd616a000,0x00007fecd626b000)]
  0x00007fed0c039800 JavaThread "http-nio-8804-exec-17" daemon [_thread_blocked, id=23050, stack(0x00007fecd626b000,0x00007fecd636c000)]
  0x00007fed0c035800 JavaThread "http-nio-8804-exec-15" daemon [_thread_blocked, id=23048, stack(0x00007fecd646d000,0x00007fecd656e000)]
  0x00007fed0c033800 JavaThread "http-nio-8804-exec-14" daemon [_thread_blocked, id=23047, stack(0x00007fecd656e000,0x00007fecd666f000)]
  0x00007fed0c031800 JavaThread "http-nio-8804-exec-13" daemon [_thread_in_Java, id=23046, stack(0x00007fecd666f000,0x00007fecd6770000)]
  0x00007fed0c023800 JavaThread "http-nio-8804-exec-9" daemon [_thread_blocked, id=23042, stack(0x00007fecd6a73000,0x00007fecd6b74000)]
  0x00007fed0c01e000 JavaThread "http-nio-8804-exec-6" daemon [_thread_in_vm, id=23039, stack(0x00007fecd6d76000,0x00007fecd6e77000)]
  0x00007fed2cb61800 JavaThread "ajp-bio-8909-AsyncTimeout" daemon [_thread_blocked, id=23034, stack(0x00007fecd727b000,0x00007fecd737c000)]
  0x00007fed2c3af800 JavaThread "ajp-bio-8909-Acceptor-0" daemon [_thread_in_native, id=23033, stack(0x00007fecd737c000,0x00007fecd747d000)]
  0x00007fed10076000 JavaThread "task-scheduler-9" [_thread_blocked, id=23032, stack(0x00007fecd747d000,0x00007fecd757e000)]
  0x00007fed10073800 JavaThread "executor-7" [_thread_blocked, id=23031, stack(0x00007fed15c4c000,0x00007fed15d4d000)]
  0x00007fed2c3ad800 JavaThread "http-nio-8804-Acceptor-0" daemon [_thread_in_native, id=23030, stack(0x00007fed15d4d000,0x00007fed15e4e000)]
  0x00007fed2c3ab800 JavaThread "http-nio-8804-ClientPoller-1" daemon [_thread_blocked, id=23029, stack(0x00007fed15e4e000,0x00007fed15f4f000)]
  0x00007fed2c3aa000 JavaThread "http-nio-8804-ClientPoller-0" daemon [_thread_in_native, id=23028, stack(0x00007fed15f4f000,0x00007fed16050000)]
  0x00007fed2c3a8000 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=23027, stack(0x00007fed16050000,0x00007fed16151000)]
  0x00007fecec037800 JavaThread "task-scheduler-8" [_thread_blocked, id=23026, stack(0x00007fecd757e000,0x00007fecd767f000)]
  0x00007fecec036800 JavaThread "executor-6" [_thread_blocked, id=23025, stack(0x00007fecd767f000,0x00007fecd7780000)]
  0x00007fed0a167800 JavaThread "task-scheduler-7" [_thread_blocked, id=23024, stack(0x00007fecd7780000,0x00007fecd7881000)]
  0x00007fed0a164000 JavaThread "executor-5" [_thread_blocked, id=23023, stack(0x00007fecd7881000,0x00007fecd7982000)]
  0x00007fed10051800 JavaThread "task-scheduler-6" [_thread_blocked, id=23022, stack(0x00007fecd7982000,0x00007fecd7a83000)]
  0x00007fed10050000 JavaThread "executor-4" [_thread_blocked, id=23021, stack(0x00007fecd7a83000,0x00007fecd7b84000)]
  0x00007fed2c3a5800 JavaThread "task-scheduler-5" [_thread_blocked, id=23020, stack(0x00007fecd7bfc000,0x00007fecd7cfd000)]
  0x00007fed2c3a4000 JavaThread "executor-3" [_thread_blocked, id=23019, stack(0x00007fecd7cfd000,0x00007fecd7dfe000)]
  0x00007fed1004e000 JavaThread "task-scheduler-4" [_thread_blocked, id=23018, stack(0x00007fecd7dfe000,0x00007fecd7eff000)]
  0x00007fed1004b800 JavaThread "executor-2" [_thread_blocked, id=23017, stack(0x00007fecd7eff000,0x00007fecd8000000)]
  0x00007fed09f6f800 JavaThread "task-scheduler-3" [_thread_blocked, id=23016, stack(0x00007fed1406b000,0x00007fed1416c000)]
  0x00007fed09f6d800 JavaThread "executor-1" [_thread_blocked, id=23015, stack(0x00007fed1416c000,0x00007fed1426d000)]
  0x00007fed09f6b800 JavaThread "task-scheduler-2" [_thread_blocked, id=23014, stack(0x00007fed1426d000,0x00007fed1436e000)]
  0x00007fed08498000 JavaThread "task-scheduler-1" [_thread_blocked, id=23013, stack(0x00007fed1436e000,0x00007fed1446f000)]
  0x00007fed09bd1000 JavaThread "Timer-1" daemon [_thread_blocked, id=23012, stack(0x00007fed1446f000,0x00007fed14570000)]
  0x00007fed0843f800 JavaThread "Timer-0" daemon [_thread_blocked, id=23011, stack(0x00007fed14671000,0x00007fed14772000)]
  0x00007fed08774000 JavaThread "Thread-24" daemon [_thread_blocked, id=23002, stack(0x00007fed14570000,0x00007fed14671000)]
  0x00007fecf0013800 JavaThread "task-scheduler-2" [_thread_blocked, id=23000, stack(0x00007fed14772000,0x00007fed14873000)]
  0x00007fed08420800 JavaThread "net.sf.ehcache.CacheManager@62e151b8" daemon [_thread_blocked, id=22092, stack(0x00007fed14873000,0x00007fed14974000)]
  0x00007fed08c40800 JavaThread "ccsCache.data" [_thread_blocked, id=21831, stack(0x00007fed14974000,0x00007fed14a75000)]
  0x00007fed08221000 JavaThread "secondLayerCache" daemon [_thread_blocked, id=21830, stack(0x00007fed14bdf000,0x00007fed14ce0000)]
  0x00007fed093e2000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bragpj9l1mz3lfthde1x5|71620343]-HelperThread-#2" daemon [_thread_blocked, id=21829, stack(0x00007fed16d6c000,0x00007f
ed16e6d000)]
  0x00007fed087b1800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bragpj9l1mz3lfthde1x5|71620343]-HelperThread-#1" daemon [_thread_blocked, id=21828, stack(0x00007fed16b6a000,0x00007f
ed16c6b000)]
  0x00007fed098eb000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bragpj9l1mz3lfthde1x5|71620343]-HelperThread-#0" daemon [_thread_blocked, id=21827, stack(0x00007fed16f6e000,0x00007f
ed1706f000)]
  0x00007fed098ea000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bragpj9l1mz3lfthde1x5|71620343]-AdminTaskTimer" daemon [_thread_blocked, id=21826, stack(0x00007fed16e6d000,0x00007fe
d16f6e000)]
  0x00007fecf4021800 JavaThread "appdynamics-analytics-writer0" daemon [_thread_blocked, id=21812, stack(0x00007fed16c6b000,0x00007fed16d6c000)]
  0x00007fed0967e000 JavaThread "task-scheduler-1" [_thread_blocked, id=21741, stack(0x00007fed153eb000,0x00007fed154ec000)]
  0x00007fed0938d000 JavaThread "net.sf.ehcache.CacheManager@7cece661" daemon [_thread_blocked, id=21601, stack(0x00007fed154ec000,0x00007fed155ed000)]
  0x00007fed091c9000 JavaThread "ccsCache.data" [_thread_blocked, id=21555, stack(0x00007fed155ed000,0x00007fed156ee000)]
  0x00007fed086a5000 JavaThread "secondLayerCache" daemon [_thread_blocked, id=21554, stack(0x00007fed156ee000,0x00007fed157ef000)]
  0x00007fed09252800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bragpj9l1mz2awtw1v7q2|5cdab40c]-HelperThread-#2" daemon [_thread_blocked, id=21493, stack(0x00007fed157ef000,0x00007f
ed158f0000)]
  0x00007fed0925c000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bragpj9l1mz2awtw1v7q2|5cdab40c]-HelperThread-#1" daemon [_thread_blocked, id=21492, stack(0x00007fed158f0000,0x00007f
ed159f1000)]
  0x00007fed09259800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bragpj9l1mz2awtw1v7q2|5cdab40c]-HelperThread-#0" daemon [_thread_blocked, id=21491, stack(0x00007fed159f1000,0x00007f
ed15af2000)]
  0x00007fed0925a800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bragpj9l1mz2awtw1v7q2|5cdab40c]-AdminTaskTimer" daemon [_thread_blocked, id=21490, stack(0x00007fed15af2000,0x00007fe
d15bf3000)]
  0x00007fed2cddb000 JavaThread "NioBlockingSelector.BlockPoller-1" daemon [_thread_blocked, id=21399, stack(0x00007fed1616c000,0x00007fed1626d000)]
  0x00007fed2cdad800 JavaThread "GC Daemon" daemon [_thread_blocked, id=21398, stack(0x00007fed165fb000,0x00007fed166fc000)]
  0x00007fed2c3b2000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=21396, stack(0x00007fed16867000,0x00007fed16968000)]
  0x00007fed2c364800 JavaThread "RMI TCP Accept-8709" daemon [_thread_in_native, id=21395, stack(0x00007fed16968000,0x00007fed16a69000)]
  0x00007fed2c331800 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=21394, stack(0x00007fed16a69000,0x00007fed16b6a000)]
  0x00007fed2c299000 JavaThread "Service Thread" daemon [_thread_blocked, id=21385, stack(0x00007fed1706f000,0x00007fed17170000)]
  0x00007fed2c298000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=21384, stack(0x00007fed17170000,0x00007fed17271000)]
  0x00007fed2c297800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=21383, stack(0x00007fed1c20d000,0x00007fed1c30e000)]
=>0x00007fecf832f000 JavaThread "AD Thread-Metric Reporter0" daemon [_thread_in_Java, id=21367, stack(0x00007fed17487000,0x00007fed17588000)]
  0x00007fecf832e000 JavaThread "AD Thread-Config Poller" daemon [_thread_blocked, id=21366, stack(0x00007fed17588000,0x00007fed17689000)]
  0x00007fecf81a3800 JavaThread "Thread-1" daemon [_thread_blocked, id=21363, stack(0x00007fed1c10c000,0x00007fed1c20d000)]
  0x00007fed2c166000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=21361, stack(0x00007fed1c33d000,0x00007fed1c43e000)]
  0x00007fed2c164000 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=21360, stack(0x00007fed1c43e000,0x00007fed1c53f000)]
  0x00007fed2c11c000 JavaThread "Finalizer" daemon [_thread_blocked, id=21359, stack(0x00007fed1c53f000,0x00007fed1c640000)]
  0x00007fed2c119800 JavaThread "Reference Handler" daemon [_thread_blocked, id=21358, stack(0x00007fed1c640000,0x00007fed1c741000)]
  0x00007fed2c00a800 JavaThread "main" [_thread_in_native, id=21353, stack(0x00007fed325f9000,0x00007fed326fa000)]

Other Threads:
  0x00007fed2c111000 VMThread [stack: 0x00007fed1c741000,0x00007fed1c842000] [id=21357]
  0x00007fed2c3b4000 WatcherThread [stack: 0x00007fed16766000,0x00007fed16867000] [id=21397]

VM state:synchronizing (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00007fed2c008480] Safepoint_lock - owner thread: 0x00007fed2c111000
[0x00007fed2c008500] Threads_lock - owner thread: 0x00007fed2c111000

Heap
 par new generation   total 2831168K, used 91431K [0x0000000660000000, 0x0000000720000000, 0x0000000720000000)
  eden space 2516608K,   0% used [0x0000000660000000, 0x0000000660b81b60, 0x00000006f99a0000)
  from space 314560K,  25% used [0x00000006f99a0000, 0x00000006fe7682b0, 0x000000070ccd0000)
  to   space 314560K,   0% used [0x000000070ccd0000, 0x000000070ccd0000, 0x0000000720000000)
 concurrent mark-sweep generation total 3145728K, used 1456475K [0x0000000720000000, 0x00000007e0000000, 0x00000007e0000000)
 concurrent-mark-sweep perm gen total 296496K, used 184519K [0x00000007e0000000, 0x00000007f218c000, 0x0000000800000000)

Card table byte_map: [0x00007fed30bba000,0x00007fed318bb000] byte_map_base: 0x00007fed2d8ba000

Polling page: 0x00007fed3364c000

Code Cache  [0x00007fed2316d000, 0x00007fed24edd000, 0x00007fed2616d000)
 total_blobs=7012 nmethods=6259 adapters=703 free_code_cache=19282Kb largest_free_block=19506624

Compilation events (10 events):
Event: 28936.084 Thread 0x00007fed2c297800 6859             org.apache.xerces.dom.ElementImpl::getAttributeNodeNS (33 bytes)
Event: 28936.118 Thread 0x00007fed2c297800 nmethod 6859 0x00007fed24eb6c90 code [0x00007fed24eb6e40, 0x00007fed24eb73d8]
Event: 28940.713 Thread 0x00007fed2c298000 6860             org.apache.xerces.dom.NodeImpl::isNormalized (56 bytes)
Event: 28940.715 Thread 0x00007fed2c298000 nmethod 6860 0x00007fed24eb2550 code [0x00007fed24eb26a0, 0x00007fed24eb2788]
Event: 28977.941 Thread 0x00007fed2c297800 6861             sun.security.util.DerInputBuffer::getBigInteger (80 bytes)
Event: 28977.994 Thread 0x00007fed2c297800 nmethod 6861 0x00007fed24eca790 code [0x00007fed24eca960, 0x00007fed24ecbd68]
Event: 29018.798 Thread 0x00007fed2c298000 6862             org.apache.http.impl.cookie.BestMatchSpec::formatCookies (109 bytes)
Event: 29018.799 Thread 0x00007fed2c297800 6863             org.apache.http.impl.cookie.BrowserCompatSpec::formatCookies (184 bytes)
Event: 29018.840 Thread 0x00007fed2c297800 nmethod 6863 0x00007fed24ecc5d0 code [0x00007fed24ecc8c0, 0x00007fed24ecdae8]
Event: 29018.897 Thread 0x00007fed2c298000 nmethod 6862 0x00007fed24ecefd0 code [0x00007fed24ecf340, 0x00007fed24ed0c58]

GC Heap History (10 events):
Event: 28747.715 GC heap before
{Heap before GC invocations=313 (full 6):
 par new generation   total 2831168K, used 2596668K [0x0000000660000000, 0x0000000720000000, 0x0000000720000000)
  eden space 2516608K, 100% used [0x0000000660000000, 0x00000006f99a0000, 0x00000006f99a0000)
  from space 314560K,  25% used [0x000000070ccd0000, 0x0000000711aff280, 0x0000000720000000)
  to   space 314560K,   0% used [0x00000006f99a0000, 0x00000006f99a0000, 0x000000070ccd0000)
 concurrent mark-sweep generation total 3145728K, used 1436443K [0x0000000720000000, 0x00000007e0000000, 0x00000007e0000000)
 concurrent-mark-sweep perm gen total 296496K, used 184512K [0x00000007e0000000, 0x00000007f218c000, 0x0000000800000000)
Event: 28747.953 GC heap after
Heap after GC invocations=314 (full 6):
 par new generation   total 2831168K, used 78436K [0x0000000660000000, 0x0000000720000000, 0x0000000720000000)
  eden space 2516608K,   0% used [0x0000000660000000, 0x0000000660000000, 0x00000006f99a0000)
  from space 314560K,  24% used [0x00000006f99a0000, 0x00000006fe639180, 0x000000070ccd0000)
  to   space 314560K,   0% used [0x000000070ccd0000, 0x000000070ccd0000, 0x0000000720000000)
 concurrent mark-sweep generation total 3145728K, used 1442723K [0x0000000720000000, 0x00000007e0000000, 0x00000007e0000000)
 concurrent-mark-sweep perm gen total 296496K, used 184512K [0x00000007e0000000, 0x00000007f218c000, 0x0000000800000000)
}
Event: 28818.124 GC heap before
{Heap before GC invocations=314 (full 6):
 par new generation   total 2831168K, used 2595044K [0x0000000660000000, 0x0000000720000000, 0x0000000720000000)
  eden space 2516608K, 100% used [0x0000000660000000, 0x00000006f99a0000, 0x00000006f99a0000)
  from space 314560K,  24% used [0x00000006f99a0000, 0x00000006fe639180, 0x000000070ccd0000)
  to   space 314560K,   0% used [0x000000070ccd0000, 0x000000070ccd0000, 0x0000000720000000)
 concurrent mark-sweep generation total 3145728K, used 1442723K [0x0000000720000000, 0x00000007e0000000, 0x00000007e0000000)
 concurrent-mark-sweep perm gen total 296496K, used 184513K [0x00000007e0000000, 0x00000007f218c000, 0x0000000800000000)
Event: 28818.316 GC heap after
Heap after GC invocations=315 (full 6):
 par new generation   total 2831168K, used 77901K [0x0000000660000000, 0x0000000720000000, 0x0000000720000000)
  eden space 2516608K,   0% used [0x0000000660000000, 0x0000000660000000, 0x00000006f99a0000)
  from space 314560K,  24% used [0x000000070ccd0000, 0x00000007118e3450, 0x0000000720000000)
  to   space 314560K,   0% used [0x00000006f99a0000, 0x00000006f99a0000, 0x000000070ccd0000)
 concurrent mark-sweep generation total 3145728K, used 1448963K [0x0000000720000000, 0x00000007e0000000, 0x00000007e0000000)
 concurrent-mark-sweep perm gen total 296496K, used 184513K [0x00000007e0000000, 0x00000007f218c000, 0x0000000800000000)
}
Event: 28918.361 GC heap before
{Heap before GC invocations=315 (full 6):
 par new generation   total 2831168K, used 2594509K [0x0000000660000000, 0x0000000720000000, 0x0000000720000000)
  eden space 2516608K, 100% used [0x0000000660000000, 0x00000006f99a0000, 0x00000006f99a0000)
  from space 314560K,  24% used [0x000000070ccd0000, 0x00000007118e3450, 0x0000000720000000)
  to   space 314560K,   0% used [0x00000006f99a0000, 0x00000006f99a0000, 0x000000070ccd0000)
 concurrent mark-sweep generation total 3145728K, used 1448963K [0x0000000720000000, 0x00000007e0000000, 0x00000007e0000000)
 concurrent-mark-sweep perm gen total 296496K, used 184513K [0x00000007e0000000, 0x00000007f218c000, 0x0000000800000000)
Event: 28918.512 GC heap after
Heap after GC invocations=316 (full 6):
 par new generation   total 2831168K, used 67070K [0x0000000660000000, 0x0000000720000000, 0x0000000720000000)
  eden space 2516608K,   0% used [0x0000000660000000, 0x0000000660000000, 0x00000006f99a0000)
  from space 314560K,  21% used [0x00000006f99a0000, 0x00000006fdb1f808, 0x000000070ccd0000)
  to   space 314560K,   0% used [0x000000070ccd0000, 0x000000070ccd0000, 0x0000000720000000)
 concurrent mark-sweep generation total 3145728K, used 1451881K [0x0000000720000000, 0x00000007e0000000, 0x00000007e0000000)
 concurrent-mark-sweep perm gen total 296496K, used 184513K [0x00000007e0000000, 0x00000007f218c000, 0x0000000800000000)
}
Event: 28970.014 GC heap before
{Heap before GC invocations=316 (full 6):
 par new generation   total 2831168K, used 2583678K [0x0000000660000000, 0x0000000720000000, 0x0000000720000000)
  eden space 2516608K, 100% used [0x0000000660000000, 0x00000006f99a0000, 0x00000006f99a0000)
  from space 314560K,  21% used [0x00000006f99a0000, 0x00000006fdb1f808, 0x000000070ccd0000)
  to   space 314560K,   0% used [0x000000070ccd0000, 0x000000070ccd0000, 0x0000000720000000)
 concurrent mark-sweep generation total 3145728K, used 1451881K [0x0000000720000000, 0x00000007e0000000, 0x00000007e0000000)
 concurrent-mark-sweep perm gen total 296496K, used 184519K [0x00000007e0000000, 0x00000007f218c000, 0x0000000800000000)
Event: 28970.187 GC heap after
Heap after GC invocations=317 (full 6):
 par new generation   total 2831168K, used 77298K [0x0000000660000000, 0x0000000720000000, 0x0000000720000000)
  eden space 2516608K,   0% used [0x0000000660000000, 0x0000000660000000, 0x00000006f99a0000)
  from space 314560K,  24% used [0x000000070ccd0000, 0x000000071184c880, 0x0000000720000000)
  to   space 314560K,   0% used [0x00000006f99a0000, 0x00000006f99a0000, 0x000000070ccd0000)
 concurrent mark-sweep generation total 3145728K, used 1453731K [0x0000000720000000, 0x00000007e0000000, 0x00000007e0000000)
 concurrent-mark-sweep perm gen total 296496K, used 184519K [0x00000007e0000000, 0x00000007f218c000, 0x0000000800000000)
}
Event: 29032.127 GC heap before
{Heap before GC invocations=317 (full 6):
 par new generation   total 2831168K, used 2593906K [0x0000000660000000, 0x0000000720000000, 0x0000000720000000)
  eden space 2516608K, 100% used [0x0000000660000000, 0x00000006f99a0000, 0x00000006f99a0000)
  from space 314560K,  24% used [0x000000070ccd0000, 0x000000071184c880, 0x0000000720000000)
  to   space 314560K,   0% used [0x00000006f99a0000, 0x00000006f99a0000, 0x000000070ccd0000)
 concurrent mark-sweep generation total 3145728K, used 1453731K [0x0000000720000000, 0x00000007e0000000, 0x00000007e0000000)
 concurrent-mark-sweep perm gen total 296496K, used 184519K [0x00000007e0000000, 0x00000007f218c000, 0x0000000800000000)
Event: 29032.306 GC heap after
Heap after GC invocations=318 (full 6):
 par new generation   total 2831168K, used 79648K [0x0000000660000000, 0x0000000720000000, 0x0000000720000000)
  eden space 2516608K,   0% used [0x0000000660000000, 0x0000000660000000, 0x00000006f99a0000)
  from space 314560K,  25% used [0x00000006f99a0000, 0x00000006fe7682b0, 0x000000070ccd0000)
  to   space 314560K,   0% used [0x000000070ccd0000, 0x000000070ccd0000, 0x0000000720000000)
 concurrent mark-sweep generation total 3145728K, used 1456475K [0x0000000720000000, 0x00000007e0000000, 0x00000007e0000000)
 concurrent-mark-sweep perm gen total 296496K, used 184519K [0x00000007e0000000, 0x00000007f218c000, 0x0000000800000000)
}

Deoptimization events (10 events):
Event: 23204.032 Thread 0x00007fed0c057800 Uncommon trap -83 fr.pc 0x00007fed24c90a64
Event: 23516.015 Thread 0x00007fed083d8800 Uncommon trap -83 fr.pc 0x00007fed24cb9b20
Event: 23797.041 Thread 0x00007fed093e1000 Uncommon trap -83 fr.pc 0x00007fed238ff078
Event: 24385.359 Thread 0x00007fed0c042000 Uncommon trap -28 fr.pc 0x00007fed240a68fc
Event: 24412.153 Thread 0x00007fecf832f000 Uncommon trap 32 fr.pc 0x00007fed2459d460
Event: 26683.435 Thread 0x00007fed08c88000 Uncommon trap -12 fr.pc 0x00007fed24e76b04
Event: 26915.168 Thread 0x00007fed08cff800 Uncommon trap -83 fr.pc 0x00007fed246bc0b4
Event: 27022.440 Thread 0x00007fed0c044800 Uncommon trap -83 fr.pc 0x00007fed23bd27fc
Event: 27022.455 Thread 0x00007fed0c044800 Uncommon trap -83 fr.pc 0x00007fed24e2c800
Event: 28036.085 Thread 0x00007fed08cff800 Uncommon trap -12 fr.pc 0x00007fed24a440dc

Internal exceptions (10 events):
Event: 29031.485 Thread 0x00007fecf4021800 Threw 0x00000006f890ad68 at /HUDSON/workspace/jdk7u21-2-build-linux-amd64-product/jdk7u21/hotspot/src/share/vm/prims/jni.cpp:742
Event: 29031.715 Thread 0x00007fed0c06c800 Threw 0x00000006ec4e93c0 at /HUDSON/workspace/jdk7u21-2-build-linux-amd64-product/jdk7u21/hotspot/src/share/vm/prims/jni.cpp:716
Event: 29031.742 Thread 0x00007fecf4021800 Threw 0x00000006f890ceb0 at /HUDSON/workspace/jdk7u21-2-build-linux-amd64-product/jdk7u21/hotspot/src/share/vm/prims/jni.cpp:742
Event: 29031.834 Thread 0x00007fecf4021800 Threw 0x00000006f890f020 at /HUDSON/workspace/jdk7u21-2-build-linux-amd64-product/jdk7u21/hotspot/src/share/vm/prims/jni.cpp:742
Event: 29031.934 Thread 0x00007fed0c039800 Threw 0x00000006f4c49e98 at /HUDSON/workspace/jdk7u21-2-build-linux-amd64-product/jdk7u21/hotspot/src/share/vm/prims/jni.cpp:716
Event: 29032.034 Thread 0x00007fecf4021800 Threw 0x00000006f8911058 at /HUDSON/workspace/jdk7u21-2-build-linux-amd64-product/jdk7u21/hotspot/src/share/vm/prims/jni.cpp:742
Event: 29032.064 Thread 0x00007fecf4021800 Threw 0x00000006f89130a8 at /HUDSON/workspace/jdk7u21-2-build-linux-amd64-product/jdk7u21/hotspot/src/share/vm/prims/jni.cpp:742
Event: 29032.085 Thread 0x00007fecf4021800 Threw 0x00000006f8915210 at /HUDSON/workspace/jdk7u21-2-build-linux-amd64-product/jdk7u21/hotspot/src/share/vm/prims/jni.cpp:742
Event: 29032.121 Thread 0x00007fecf4021800 Threw 0x00000006f8917640 at /HUDSON/workspace/jdk7u21-2-build-linux-amd64-product/jdk7u21/hotspot/src/share/vm/prims/jni.cpp:742
Event: 29032.326 Thread 0x00007fecf4021800 Threw 0x00000006605e13c0 at /HUDSON/workspace/jdk7u21-2-build-linux-amd64-product/jdk7u21/hotspot/src/share/vm/prims/jni.cpp:742

Events (10 events):
Event: 29020.409 Executing VM operation: RevokeBias
Event: 29020.411 Executing VM operation: RevokeBias done
Event: 29025.485 Thread 0x00007fece1b8b000 Thread added: 0x00007fece1b8b000
Event: 29025.646 Thread 0x00007fece1b8b000 Thread exited: 0x00007fece1b8b000
Event: 29028.545 Executing VM operation: RevokeBias
Event: 29028.552 Executing VM operation: RevokeBias done
Event: 29031.730 Executing VM operation: RevokeBias
Event: 29031.741 Executing VM operation: RevokeBias done
Event: 29032.126 Executing VM operation: GenCollectForAllocation
Event: 29032.306 Executing VM operation: GenCollectForAllocation done


Dynamic libraries:
00400000-00401000 r-xp 00000000 fd:0a 7438                               /usr/java/jre1.7.0_21/bin/java
00600000-00601000 rw-p 00000000 fd:0a 7438                               /usr/java/jre1.7.0_21/bin/java
01cad000-023c1000 rw-p 00000000 00:00 0                                  [heap]
660000000-7f218c000 rw-p 00000000 00:00 0
7f218c000-800000000 rw-p 00000000 00:00 0
343d600000-343d620000 r-xp 00000000 fd:01 199                            /lib64/ld-2.12.so
343d81f000-343d821000 r--p 0001f000 fd:01 199                            /lib64/ld-2.12.so
343d821000-343d822000 rw-p 00021000 fd:01 199                            /lib64/ld-2.12.so
343d822000-343d823000 rw-p 00000000 00:00 0
343da00000-343db8a000 r-xp 00000000 fd:01 418                            /lib64/libc-2.12.so
343db8a000-343dd8a000 ---p 0018a000 fd:01 418                            /lib64/libc-2.12.so
343dd8a000-343dd8e000 r--p 0018a000 fd:01 418                            /lib64/libc-2.12.so
343dd8e000-343dd90000 rw-p 0018e000 fd:01 418                            /lib64/libc-2.12.so
343dd90000-343dd94000 rw-p 00000000 00:00 0
343de00000-343de17000 r-xp 00000000 fd:01 534                            /lib64/libpthread-2.12.so
343de17000-343e017000 ---p 00017000 fd:01 534                            /lib64/libpthread-2.12.so
343e017000-343e018000 r--p 00017000 fd:01 534                            /lib64/libpthread-2.12.so
343e018000-343e019000 rw-p 00018000 fd:01 534                            /lib64/libpthread-2.12.so
343e019000-343e01d000 rw-p 00000000 00:00 0
343e200000-343e202000 r-xp 00000000 fd:01 592                            /lib64/libdl-2.12.so
343e202000-343e402000 ---p 00002000 fd:01 592                            /lib64/libdl-2.12.so
343e402000-343e403000 r--p 00002000 fd:01 592                            /lib64/libdl-2.12.so
343e403000-343e404000 rw-p 00003000 fd:01 592                            /lib64/libdl-2.12.so
343e600000-343e683000 r-xp 00000000 fd:01 602                            /lib64/libm-2.12.so
343e683000-343e882000 ---p 00083000 fd:01 602                            /lib64/libm-2.12.so
343e882000-343e883000 r--p 00082000 fd:01 602                            /lib64/libm-2.12.so
343e883000-343e884000 rw-p 00083000 fd:01 602                            /lib64/libm-2.12.so
343ee00000-343ee07000 r-xp 00000000 fd:01 614                            /lib64/librt-2.12.so
343ee07000-343f006000 ---p 00007000 fd:01 614                            /lib64/librt-2.12.so
343f006000-343f007000 r--p 00006000 fd:01 614                            /lib64/librt-2.12.so
343f007000-343f008000 rw-p 00007000 fd:01 614                            /lib64/librt-2.12.so
343fa00000-343fa16000 r-xp 00000000 fd:01 607                            /lib64/libgcc_s-4.4.7-20120601.so.1
343fa16000-343fc15000 ---p 00016000 fd:01 607                            /lib64/libgcc_s-4.4.7-20120601.so.1
343fc15000-343fc16000 rw-p 00015000 fd:01 607                            /lib64/libgcc_s-4.4.7-20120601.so.1
343fe00000-343fe16000 r-xp 00000000 fd:01 835                            /lib64/libresolv-2.12.so
343fe16000-3440016000 ---p 00016000 fd:01 835                            /lib64/libresolv-2.12.so
3440016000-3440017000 r--p 00016000 fd:01 835                            /lib64/libresolv-2.12.so
3440017000-3440018000 rw-p 00017000 fd:01 835                            /lib64/libresolv-2.12.so
3440018000-344001a000 rw-p 00000000 00:00 0
7feccc745000-7feccc748000 ---p 00000000 00:00 0
7feccc748000-7feccc846000 rw-p 00000000 00:00 0
7feccc846000-7feccc849000 ---p 00000000 00:00 0
7feccc849000-7feccc947000 rw-p 00000000 00:00 0
7feccc947000-7feccc94a000 ---p 00000000 00:00 0
7feccc94a000-7feccca48000 rw-p 00000000 00:00 0
7fecccdce000-7fecccdd1000 ---p 00000000 00:00 0
7fecccdd1000-7fecccecf000 rw-p 00000000 00:00 0
7fecccecf000-7feccced2000 ---p 00000000 00:00 0
7feccced2000-7fecccfd0000 rw-p 00000000 00:00 0
7fecccfd0000-7fecccfd3000 ---p 00000000 00:00 0
7fecccfd3000-7feccd0d1000 rw-p 00000000 00:00 0
7feccd0d1000-7feccd0d4000 ---p 00000000 00:00 0
7feccd0d4000-7feccd1d2000 rw-p 00000000 00:00 0
7feccd1d2000-7feccd1d5000 ---p 00000000 00:00 0
7feccd1d5000-7feccd2d3000 rw-p 00000000 00:00 0
7feccd2d3000-7feccd2d6000 ---p 00000000 00:00 0
7feccd2d6000-7feccd3d4000 rw-p 00000000 00:00 0
7feccd3d4000-7feccd3d7000 ---p 00000000 00:00 0
7feccd3d7000-7feccd4d5000 rw-p 00000000 00:00 0
7feccd4d5000-7feccd4d8000 ---p 00000000 00:00 0
7feccd4d8000-7feccd5d6000 rw-p 00000000 00:00 0
7feccd5d6000-7feccd5d9000 ---p 00000000 00:00 0
7feccd5d9000-7feccd6d7000 rw-p 00000000 00:00 0
7feccd6d7000-7feccd6da000 ---p 00000000 00:00 0
7feccd6da000-7feccd7d8000 rw-p 00000000 00:00 0
7feccd7d8000-7feccd7db000 ---p 00000000 00:00 0
7feccd7db000-7feccd8d9000 rw-p 00000000 00:00 0
7feccd8d9000-7feccd8dc000 ---p 00000000 00:00 0
7feccd8dc000-7feccd9da000 rw-p 00000000 00:00 0
7feccd9da000-7feccd9dd000 ---p 00000000 00:00 0
7feccd9dd000-7feccdadb000 rw-p 00000000 00:00 0
7feccdadb000-7feccdade000 ---p 00000000 00:00 0
7feccdade000-7feccdbdc000 rw-p 00000000 00:00 0
7feccdbdc000-7feccdbdf000 ---p 00000000 00:00 0
7feccdbdf000-7feccdcdd000 rw-p 00000000 00:00 0
7feccdcdd000-7feccdce0000 ---p 00000000 00:00 0
7feccdce0000-7feccddde000 rw-p 00000000 00:00 0
7feccddde000-7feccdde1000 ---p 00000000 00:00 0
7feccdde1000-7feccdedf000 rw-p 00000000 00:00 0
7feccdedf000-7feccdee2000 ---p 00000000 00:00 0
7feccdee2000-7feccdfe0000 rw-p 00000000 00:00 0
7feccdfe0000-7feccdfe3000 ---p 00000000 00:00 0
7feccdfe3000-7fecce0e1000 rw-p 00000000 00:00 0
7fecce0e1000-7fecce0e4000 ---p 00000000 00:00 0
7fecce0e4000-7fecce1e2000 rw-p 00000000 00:00 0
7fecce1e2000-7fecce1e5000 ---p 00000000 00:00 0
7fecce1e5000-7fecce2e3000 rw-p 00000000 00:00 0
7fecce2e3000-7fecce2e6000 ---p 00000000 00:00 0
7fecce2e6000-7fecce3e4000 rw-p 00000000 00:00 0
7fecce3e4000-7fecce3e7000 ---p 00000000 00:00 0
7fecce3e7000-7fecce4e5000 rw-p 00000000 00:00 0
7fecce4e5000-7fecce4e8000 ---p 00000000 00:00 0
7fecce4e8000-7fecce5e6000 rw-p 00000000 00:00 0
7fecce5e6000-7fecce5e9000 ---p 00000000 00:00 0
7fecce5e9000-7fecce6e7000 rw-p 00000000 00:00 0
7fecce6e7000-7fecce6ea000 ---p 00000000 00:00 0
7fecce6ea000-7fecce7e8000 rw-p 00000000 00:00 0
7fecce7e8000-7fecce7eb000 ---p 00000000 00:00 0
7fecce7eb000-7fecce8e9000 rw-p 00000000 00:00 0
7fecce8e9000-7fecce8ec000 ---p 00000000 00:00 0
7fecce8ec000-7fecce9ea000 rw-p 00000000 00:00 0
7fecce9ea000-7fecce9ed000 ---p 00000000 00:00 0
7fecce9ed000-7fecceaeb000 rw-p 00000000 00:00 0
7fecceaeb000-7fecceaee000 ---p 00000000 00:00 0
7fecceaee000-7feccebec000 rw-p 00000000 00:00 0
7feccebec000-7feccebef000 ---p 00000000 00:00 0
7feccebef000-7fecceced000 rw-p 00000000 00:00 0
7fecceced000-7feccecf0000 ---p 00000000 00:00 0
7feccecf0000-7feccedee000 rw-p 00000000 00:00 0
7fecceeef000-7fecceef2000 ---p 00000000 00:00 0
7fecceef2000-7fecceff0000 rw-p 00000000 00:00 0
7fecceff0000-7fecceff3000 ---p 00000000 00:00 0
7fecceff3000-7feccf0f1000 rw-p 00000000 00:00 0
7feccf0f1000-7feccf0f4000 ---p 00000000 00:00 0
7feccf0f4000-7feccf1f2000 rw-p 00000000 00:00 0
7feccf1f2000-7feccf1f5000 ---p 00000000 00:00 0
7feccf1f5000-7feccf2f3000 rw-p 00000000 00:00 0
7feccf2f3000-7feccf2f6000 ---p 00000000 00:00 0
7feccf2f6000-7feccf3f4000 rw-p 00000000 00:00 0
7feccf3f4000-7feccf3f7000 ---p 00000000 00:00 0
7feccf3f7000-7feccf4f5000 rw-p 00000000 00:00 0
7feccf4f5000-7feccf4f8000 ---p 00000000 00:00 0
7feccf4f8000-7feccf5f6000 rw-p 00000000 00:00 0
7feccf5f6000-7feccf5f9000 ---p 00000000 00:00 0
7feccf5f9000-7feccf6f7000 rw-p 00000000 00:00 0
7feccf8f9000-7feccf8fc000 ---p 00000000 00:00 0
7feccf8fc000-7feccf9fa000 rw-p 00000000 00:00 0
7feccf9fa000-7feccf9fd000 ---p 00000000 00:00 0
7feccf9fd000-7feccfafb000 rw-p 00000000 00:00 0
7feccfafb000-7feccfafe000 ---p 00000000 00:00 0
7feccfafe000-7feccfbfc000 rw-p 00000000 00:00 0
7feccfbfc000-7feccfbff000 ---p 00000000 00:00 0
7feccfbff000-7feccfcfd000 rw-p 00000000 00:00 0
7feccfcfd000-7feccfd00000 ---p 00000000 00:00 0
7feccfd00000-7feccfdfe000 rw-p 00000000 00:00 0
7feccfdfe000-7feccfe01000 ---p 00000000 00:00 0
7feccfe01000-7feccfeff000 rw-p 00000000 00:00 0
7feccfeff000-7feccff02000 ---p 00000000 00:00 0
7feccff02000-7fecd0000000 rw-p 00000000 00:00 0
7fecd0000000-7fecd0e41000 rw-p 00000000 00:00 0
7fecd0e41000-7fecd4000000 ---p 00000000 00:00 0
7fecd4049000-7fecd404c000 ---p 00000000 00:00 0
7fecd404c000-7fecd414a000 rw-p 00000000 00:00 0
7fecd414a000-7fecd414d000 ---p 00000000 00:00 0
7fecd414d000-7fecd424b000 rw-p 00000000 00:00 0
7fecd424b000-7fecd424e000 ---p 00000000 00:00 0
7fecd424e000-7fecd434c000 rw-p 00000000 00:00 0
7fecd434c000-7fecd434f000 ---p 00000000 00:00 0
7fecd434f000-7fecd444d000 rw-p 00000000 00:00 0
7fecd444d000-7fecd4450000 ---p 00000000 00:00 0
7fecd4450000-7fecd454e000 rw-p 00000000 00:00 0
7fecd454e000-7fecd4551000 ---p 00000000 00:00 0
7fecd4551000-7fecd464f000 rw-p 00000000 00:00 0
7fecd464f000-7fecd4652000 ---p 00000000 00:00 0
7fecd4652000-7fecd4750000 rw-p 00000000 00:00 0
7fecd4750000-7fecd4753000 ---p 00000000 00:00 0
7fecd4753000-7fecd4851000 rw-p 00000000 00:00 0
7fecd4851000-7fecd4854000 ---p 00000000 00:00 0
7fecd4854000-7fecd4952000 rw-p 00000000 00:00 0
7fecd4952000-7fecd4955000 ---p 00000000 00:00 0
7fecd4955000-7fecd4a53000 rw-p 00000000 00:00 0
7fecd4a53000-7fecd4a56000 ---p 00000000 00:00 0
7fecd4a56000-7fecd4b54000 rw-p 00000000 00:00 0
7fecd4b54000-7fecd4b57000 ---p 00000000 00:00 0
7fecd4b57000-7fecd4c55000 rw-p 00000000 00:00 0
7fecd4c55000-7fecd4c58000 ---p 00000000 00:00 0
7fecd4c58000-7fecd4d56000 rw-p 00000000 00:00 0
7fecd4d56000-7fecd4d59000 ---p 00000000 00:00 0
7fecd4d59000-7fecd4e57000 rw-p 00000000 00:00 0
7fecd4e57000-7fecd4e5a000 ---p 00000000 00:00 0
7fecd4e5a000-7fecd4f58000 rw-p 00000000 00:00 0
7fecd4f58000-7fecd4f5b000 ---p 00000000 00:00 0
7fecd4f5b000-7fecd5059000 rw-p 00000000 00:00 0
7fecd5059000-7fecd505c000 ---p 00000000 00:00 0
7fecd505c000-7fecd515a000 rw-p 00000000 00:00 0
7fecd515a000-7fecd515d000 ---p 00000000 00:00 0
7fecd515d000-7fecd525b000 rw-p 00000000 00:00 0
7fecd525b000-7fecd525e000 ---p 00000000 00:00 0
7fecd525e000-7fecd535c000 rw-p 00000000 00:00 0
7fecd535c000-7fecd535f000 ---p 00000000 00:00 0
7fecd535f000-7fecd545d000 rw-p 00000000 00:00 0
7fecd545d000-7fecd5460000 ---p 00000000 00:00 0
7fecd5460000-7fecd555e000 rw-p 00000000 00:00 0
7fecd565f000-7fecd5662000 ---p 00000000 00:00 0
7fecd5662000-7fecd5760000 rw-p 00000000 00:00 0
7fecd5760000-7fecd5763000 ---p 00000000 00:00 0
7fecd5763000-7fecd5861000 rw-p 00000000 00:00 0
7fecd5962000-7fecd5965000 ---p 00000000 00:00 0
7fecd5965000-7fecd5a63000 rw-p 00000000 00:00 0
7fecd5a63000-7fecd5a66000 ---p 00000000 00:00 0
7fecd5a66000-7fecd5b64000 rw-p 00000000 00:00 0
7fecd5b64000-7fecd5b67000 ---p 00000000 00:00 0
7fecd5b67000-7fecd5c65000 rw-p 00000000 00:00 0
7fecd5c65000-7fecd5c68000 ---p 00000000 00:00 0
7fecd5c68000-7fecd5d66000 rw-p 00000000 00:00 0
7fecd5d66000-7fecd5d69000 ---p 00000000 00:00 0
7fecd5d69000-7fecd5e67000 rw-p 00000000 00:00 0
7fecd5e67000-7fecd5e6a000 ---p 00000000 00:00 0
7fecd5e6a000-7fecd5f68000 rw-p 00000000 00:00 0
7fecd5f68000-7fecd5f6b000 ---p 00000000 00:00 0
7fecd5f6b000-7fecd6069000 rw-p 00000000 00:00 0
7fecd6069000-7fecd606c000 ---p 00000000 00:00 0
7fecd606c000-7fecd616a000 rw-p 00000000 00:00 0
7fecd616a000-7fecd616d000 ---p 00000000 00:00 0
7fecd616d000-7fecd626b000 rw-p 00000000 00:00 0
7fecd626b000-7fecd626e000 ---p 00000000 00:00 0
7fecd626e000-7fecd636c000 rw-p 00000000 00:00 0
7fecd636c000-7fecd636f000 ---p 00000000 00:00 0
7fecd636f000-7fecd646d000 rw-p 00000000 00:00 0
7fecd646d000-7fecd6470000 ---p 00000000 00:00 0
7fecd6470000-7fecd656e000 rw-p 00000000 00:00 0
7fecd656e000-7fecd6571000 ---p 00000000 00:00 0
7fecd6571000-7fecd666f000 rw-p 00000000 00:00 0
7fecd666f000-7fecd6672000 ---p 00000000 00:00 0
7fecd6672000-7fecd6770000 rw-p 00000000 00:00 0
7fecd6770000-7fecd6773000 ---p 00000000 00:00 0
7fecd6773000-7fecd6871000 rw-p 00000000 00:00 0
7fecd6871000-7fecd6874000 ---p 00000000 00:00 0
7fecd6874000-7fecd6972000 rw-p 00000000 00:00 0
7fecd6972000-7fecd6975000 ---p 00000000 00:00 0
7fecd6975000-7fecd6a73000 rw-p 00000000 00:00 0
7fecd6a73000-7fecd6a76000 ---p 00000000 00:00 0
7fecd6a76000-7fecd6b74000 rw-p 00000000 00:00 0
7fecd6b74000-7fecd6b77000 ---p 00000000 00:00 0
7fecd6b77000-7fecd6c75000 rw-p 00000000 00:00 0
7fecd6c75000-7fecd6c78000 ---p 00000000 00:00 0
7fecd6c78000-7fecd6d76000 rw-p 00000000 00:00 0
7fecd6d76000-7fecd6d79000 ---p 00000000 00:00 0
7fecd6d79000-7fecd6e77000 rw-p 00000000 00:00 0
7fecd6e77000-7fecd6e7a000 ---p 00000000 00:00 0
7fecd6e7a000-7fecd6f78000 rw-p 00000000 00:00 0
7fecd7079000-7fecd707c000 ---p 00000000 00:00 0
7fecd707c000-7fecd717a000 rw-p 00000000 00:00 0
7fecd717a000-7fecd717d000 ---p 00000000 00:00 0
7fecd717d000-7fecd727b000 rw-p 00000000 00:00 0
7fecd727b000-7fecd727e000 ---p 00000000 00:00 0
7fecd727e000-7fecd737c000 rw-p 00000000 00:00 0
7fecd737c000-7fecd737f000 ---p 00000000 00:00 0
7fecd737f000-7fecd747d000 rw-p 00000000 00:00 0
7fecd747d000-7fecd7480000 ---p 00000000 00:00 0
7fecd7480000-7fecd757e000 rw-p 00000000 00:00 0
7fecd757e000-7fecd7581000 ---p 00000000 00:00 0
7fecd7581000-7fecd767f000 rw-p 00000000 00:00 0
7fecd767f000-7fecd7682000 ---p 00000000 00:00 0
7fecd7682000-7fecd7780000 rw-p 00000000 00:00 0
7fecd7780000-7fecd7783000 ---p 00000000 00:00 0
7fecd7783000-7fecd7881000 rw-p 00000000 00:00 0
7fecd7881000-7fecd7884000 ---p 00000000 00:00 0
7fecd7884000-7fecd7982000 rw-p 00000000 00:00 0
7fecd7982000-7fecd7985000 ---p 00000000 00:00 0
7fecd7985000-7fecd7a83000 rw-p 00000000 00:00 0
7fecd7a83000-7fecd7a86000 ---p 00000000 00:00 0
7fecd7a86000-7fecd7b84000 rw-p 00000000 00:00 0
7fecd7bfc000-7fecd7bff000 ---p 00000000 00:00 0
7fecd7bff000-7fecd7cfd000 rw-p 00000000 00:00 0
7fecd7cfd000-7fecd7d00000 ---p 00000000 00:00 0
7fecd7d00000-7fecd7dfe000 rw-p 00000000 00:00 0
7fecd7dfe000-7fecd7e01000 ---p 00000000 00:00 0
7fecd7e01000-7fecd7eff000 rw-p 00000000 00:00 0
7fecd7eff000-7fecd7f02000 ---p 00000000 00:00 0
7fecd7f02000-7fecd8000000 rw-p 00000000 00:00 0
7fecd8000000-7fecdb45a000 rw-p 00000000 00:00 0
7fecdb45a000-7fecdc000000 ---p 00000000 00:00 0
7fecdc000000-7fece0000000 rw-p 00000000 00:00 0
7fece0000000-7fece3ff9000 rw-p 00000000 00:00 0
7fece3ff9000-7fece4000000 ---p 00000000 00:00 0
7fece4000000-7fece421a000 rw-p 00000000 00:00 0
7fece421a000-7fece8000000 ---p 00000000 00:00 0
7fece8000000-7fece84f6000 rw-p 00000000 00:00 0
7fece84f6000-7fecec000000 ---p 00000000 00:00 0
7fecec000000-7fecec51f000 rw-p 00000000 00:00 0
7fecec51f000-7fecf0000000 ---p 00000000 00:00 0
7fecf0000000-7fecf050c000 rw-p 00000000 00:00 0
7fecf050c000-7fecf4000000 ---p 00000000 00:00 0
7fecf4000000-7fecf4517000 rw-p 00000000 00:00 0
7fecf4517000-7fecf8000000 ---p 00000000 00:00 0
7fecf8000000-7fecfbffb000 rw-p 00000000 00:00 0
7fecfbffb000-7fecfc000000 ---p 00000000 00:00 0
7fecfc000000-7fecfc1ad000 rw-p 00000000 00:00 0
7fecfc1ad000-7fed00000000 ---p 00000000 00:00 0
7fed00000000-7fed00160000 rw-p 00000000 00:00 0
7fed00160000-7fed04000000 ---p 00000000 00:00 0
7fed04000000-7fed0480d000 rw-p 00000000 00:00 0
7fed0480d000-7fed08000000 ---p 00000000 00:00 0
7fed08000000-7fed0a1ee000 rw-p 00000000 00:00 0
7fed0a1ee000-7fed0c000000 ---p 00000000 00:00 0
7fed0c000000-7fed0c2de000 rw-p 00000000 00:00 0
7fed0c2de000-7fed10000000 ---p 00000000 00:00 0
7fed10000000-7fed108b3000 rw-p 00000000 00:00 0
7fed108b3000-7fed14000000 ---p 00000000 00:00 0
7fed1406b000-7fed1406e000 ---p 00000000 00:00 0
7fed1406e000-7fed1416c000 rw-p 00000000 00:00 0
7fed1416c000-7fed1416f000 ---p 00000000 00:00 0
7fed1416f000-7fed1426d000 rw-p 00000000 00:00 0
7fed1426d000-7fed14270000 ---p 00000000 00:00 0
7fed14270000-7fed1436e000 rw-p 00000000 00:00 0
7fed1436e000-7fed14371000 ---p 00000000 00:00 0
7fed14371000-7fed1446f000 rw-p 00000000 00:00 0
7fed1446f000-7fed14472000 ---p 00000000 00:00 0
7fed14472000-7fed14570000 rw-p 00000000 00:00 0
7fed14570000-7fed14573000 ---p 00000000 00:00 0
7fed14573000-7fed14671000 rw-p 00000000 00:00 0
7fed14671000-7fed14674000 ---p 00000000 00:00 0
7fed14674000-7fed14772000 rw-p 00000000 00:00 0
7fed14772000-7fed14775000 ---p 00000000 00:00 0
7fed14775000-7fed14873000 rw-p 00000000 00:00 0
7fed14873000-7fed14876000 ---p 00000000 00:00 0
7fed14876000-7fed14974000 rw-p 00000000 00:00 0
7fed14974000-7fed14977000 ---p 00000000 00:00 0
7fed14977000-7fed14ba9000 rw-p 00000000 00:00 0
7fed14bdf000-7fed14be2000 ---p 00000000 00:00 0
7fed14be2000-7fed14ce0000 rw-p 00000000 00:00 0
7fed153eb000-7fed153ee000 ---p 00000000 00:00 0
7fed153ee000-7fed154ec000 rw-p 00000000 00:00 0
7fed154ec000-7fed154ef000 ---p 00000000 00:00 0
7fed154ef000-7fed155ed000 rw-p 00000000 00:00 0
7fed155ed000-7fed155f0000 ---p 00000000 00:00 0
7fed155f0000-7fed156ee000 rw-p 00000000 00:00 0
7fed156ee000-7fed156f1000 ---p 00000000 00:00 0
7fed156f1000-7fed157ef000 rw-p 00000000 00:00 0
7fed157ef000-7fed157f2000 ---p 00000000 00:00 0
7fed157f2000-7fed158f0000 rw-p 00000000 00:00 0
7fed158f0000-7fed158f3000 ---p 00000000 00:00 0
7fed158f3000-7fed159f1000 rw-p 00000000 00:00 0
7fed159f1000-7fed159f4000 ---p 00000000 00:00 0
7fed159f4000-7fed15af2000 rw-p 00000000 00:00 0
7fed15af2000-7fed15af5000 ---p 00000000 00:00 0
7fed15af5000-7fed15bf3000 rw-p 00000000 00:00 0
7fed15c4c000-7fed15c4f000 ---p 00000000 00:00 0
7fed15c4f000-7fed15d4d000 rw-p 00000000 00:00 0
7fed15d4d000-7fed15d50000 ---p 00000000 00:00 0
7fed15d50000-7fed15e4e000 rw-p 00000000 00:00 0
7fed15e4e000-7fed15e51000 ---p 00000000 00:00 0
7fed15e51000-7fed15f4f000 rw-p 00000000 00:00 0
7fed15f4f000-7fed15f52000 ---p 00000000 00:00 0
7fed15f52000-7fed16050000 rw-p 00000000 00:00 0
7fed16050000-7fed16053000 ---p 00000000 00:00 0
7fed16053000-7fed16151000 rw-p 00000000 00:00 0
7fed16151000-7fed1615c000 r--s 00358000 fd:0a 7684                       /usr/java/jre1.7.0_21/lib/charsets.jar
7fed1615c000-7fed16167000 r--s 00358000 fd:0a 7684                       /usr/java/jre1.7.0_21/lib/charsets.jar
7fed1616c000-7fed1616f000 ---p 00000000 00:00 0
7fed1616f000-7fed1626d000 rw-p 00000000 00:00 0
7fed1631e000-7fed16321000 ---p 00000000 00:00 0
7fed16321000-7fed1641f000 rw-p 00000000 00:00 0
7fed16520000-7fed16523000 r--s 00029000 fd:04 393269                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/servlet-api.jar
7fed165fb000-7fed165fe000 ---p 00000000 00:00 0
7fed165fe000-7fed166fc000 rw-p 00000000 00:00 0
7fed166fc000-7fed166ff000 r--s 000f5000 fd:0a 7516                       /usr/java/jre1.7.0_21/lib/ext/localedata.jar
7fed166ff000-7fed1670c000 r--s 000b6000 fd:04 393270                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/tomcat-coyote.jar
7fed1670c000-7fed16725000 r--s 00345000 fd:04 393264                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/db2jcc4.jar
7fed16725000-7fed16727000 r--s 00011000 fd:04 393266                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/tomcat-i18n-es.jar
7fed16727000-7fed16739000 r--s 0016c000 fd:04 393265                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/catalina.jar
7fed16739000-7fed1673d000 r--s 00053000 fd:04 393267                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/tibjms.jar
7fed1673d000-7fed1673f000 r--s 0000a000 fd:04 393263                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/tomcat-i18n-fr.jar
7fed1673f000-7fed16741000 r--s 0000b000 fd:04 393280                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/tomcat-i18n-ja.jar
7fed16741000-7fed16742000 r--s 00003000 fd:04 393268                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/annotations-api.jar
7fed16742000-7fed16746000 r--s 00036000 fd:04 393256                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/tomcat-dbcp.jar
7fed16746000-7fed16749000 r--s 0001c000 fd:04 393254                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/jasper-el.jar
7fed16749000-7fed1674b000 r--s 00005000 fd:04 393252                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/jms.jar
7fed1674b000-7fed1674d000 r--s 0000a000 fd:04 393255                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/el-api.jar
7fed1674d000-7fed1675c000 r--s 001a8000 fd:04 393259                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/ecj-4.2.1.jar
7fed1675c000-7fed1675f000 r--s 00013000 fd:04 393257                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/jsp-api.jar
7fed1675f000-7fed16766000 r--s 00048000 fd:04 393261                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/tibcrypt.jar
7fed16766000-7fed16767000 ---p 00000000 00:00 0
7fed16767000-7fed16867000 rw-p 00000000 00:00 0
7fed16867000-7fed1686a000 ---p 00000000 00:00 0
7fed1686a000-7fed16968000 rw-p 00000000 00:00 0
7fed16968000-7fed1696b000 ---p 00000000 00:00 0
7fed1696b000-7fed16a69000 rw-p 00000000 00:00 0
7fed16a69000-7fed16a6c000 ---p 00000000 00:00 0
7fed16a6c000-7fed16b6a000 rw-p 00000000 00:00 0
7fed16b6a000-7fed16b6d000 ---p 00000000 00:00 0
7fed16b6d000-7fed16c6b000 rw-p 00000000 00:00 0
7fed16c6b000-7fed16c6e000 ---p 00000000 00:00 0
7fed16c6e000-7fed16d6c000 rw-p 00000000 00:00 0
7fed16d6c000-7fed16d6f000 ---p 00000000 00:00 0
7fed16d6f000-7fed16e6d000 rw-p 00000000 00:00 0
7fed16e6d000-7fed16e70000 ---p 00000000 00:00 0
7fed16e70000-7fed16f6e000 rw-p 00000000 00:00 0
7fed16f6e000-7fed16f71000 ---p 00000000 00:00 0
7fed16f71000-7fed1706f000 rw-p 00000000 00:00 0
7fed1706f000-7fed17072000 ---p 00000000 00:00 0
7fed17072000-7fed17170000 rw-p 00000000 00:00 0
7fed17170000-7fed17173000 ---p 00000000 00:00 0
7fed17173000-7fed17271000 rw-p 00000000 00:00 0
7fed17271000-7fed17276000 r-xp 00000000 fd:01 1853                       /lib64/libnss_dns-2.12.so
7fed17276000-7fed17475000 ---p 00005000 fd:01 1853                       /lib64/libnss_dns-2.12.so
7fed17475000-7fed17476000 r--p 00004000 fd:01 1853                       /lib64/libnss_dns-2.12.so
7fed17476000-7fed17477000 rw-p 00005000 fd:01 1853                       /lib64/libnss_dns-2.12.so
7fed17477000-7fed1747c000 r--s 0003a000 fd:04 393281                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/catalina-tribes.jar
7fed1747c000-7fed1747f000 r--s 0001c000 fd:04 393286                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/tomcat-jdbc.jar
7fed1747f000-7fed17487000 r--s 0008b000 fd:04 393262                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/jasper.jar
7fed17487000-7fed1748a000 ---p 00000000 00:00 0
7fed1748a000-7fed17588000 rw-p 00000000 00:00 0
7fed17588000-7fed1758b000 ---p 00000000 00:00 0
7fed1758b000-7fed17689000 rw-p 00000000 00:00 0
7fed17689000-7fed1768c000 ---p 00000000 00:00 0
7fed1768c000-7fed1778a000 rw-p 00000000 00:00 0
7fed1778a000-7fed177bc000 r-xp 00000000 fd:0a 7503                       /usr/java/jre1.7.0_21/lib/amd64/libsunec.so
7fed177bc000-7fed179bb000 ---p 00032000 fd:0a 7503                       /usr/java/jre1.7.0_21/lib/amd64/libsunec.so
7fed179bb000-7fed179c1000 rw-p 00031000 fd:0a 7503                       /usr/java/jre1.7.0_21/lib/amd64/libsunec.so
7fed179c1000-7fed179d1000 rw-p 00000000 00:00 0
7fed179d1000-7fed179d9000 r-xp 00000000 fd:0a 7493                       /usr/java/jre1.7.0_21/lib/amd64/libmanagement.so
7fed179d9000-7fed17bd8000 ---p 00008000 fd:0a 7493                       /usr/java/jre1.7.0_21/lib/amd64/libmanagement.so
7fed17bd8000-7fed17bd9000 rw-p 00007000 fd:0a 7493                       /usr/java/jre1.7.0_21/lib/amd64/libmanagement.so
7fed17bd9000-7fed17bee000 r-xp 00000000 fd:0a 7495                       /usr/java/jre1.7.0_21/lib/amd64/libnet.so
7fed17bee000-7fed17dee000 ---p 00015000 fd:0a 7495                       /usr/java/jre1.7.0_21/lib/amd64/libnet.so
7fed17dee000-7fed17def000 rw-p 00015000 fd:0a 7495                       /usr/java/jre1.7.0_21/lib/amd64/libnet.so
7fed17def000-7fed17dff000 r-xp 00000000 fd:0a 7496                       /usr/java/jre1.7.0_21/lib/amd64/libnio.so
7fed17dff000-7fed17fff000 ---p 00010000 fd:0a 7496                       /usr/java/jre1.7.0_21/lib/amd64/libnio.so
7fed17fff000-7fed18000000 rw-p 00010000 fd:0a 7496                       /usr/java/jre1.7.0_21/lib/amd64/libnio.so
7fed18000000-7fed185c7000 rw-p 00000000 00:00 0
7fed185c7000-7fed1c000000 ---p 00000000 00:00 0
7fed1c001000-7fed1c002000 r--s 00005000 fd:04 393288                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/tomcat-util.jar
7fed1c002000-7fed1c003000 r--s 00001000 fd:04 393253                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/tomcat-api.jar
7fed1c003000-7fed1c005000 r--s 0000c000 fd:04 393258                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/catalina-ant.jar
7fed1c005000-7fed1c008000 r--s 0001e000 fd:04 393260                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/lib/catalina-ha.jar
7fed1c008000-7fed1c00b000 ---p 00000000 00:00 0
7fed1c00b000-7fed1c109000 rw-p 00000000 00:00 0
7fed1c109000-7fed1c10c000 r--s 0002d000 fd:0a 7637                       /usr/java/jre1.7.0_21/lib/ext/sunjce_provider.jar
7fed1c10c000-7fed1c10f000 ---p 00000000 00:00 0
7fed1c10f000-7fed1c20d000 rw-p 00000000 00:00 0
7fed1c20d000-7fed1c210000 ---p 00000000 00:00 0
7fed1c210000-7fed1c30e000 rw-p 00000000 00:00 0
7fed1c30e000-7fed1c319000 r--s 00085000 fd:04 131167                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/httpclient-4.3.3.jar
7fed1c319000-7fed1c33d000 r--s 00154000 fd:04 131174                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/singularity-xml.jar
7fed1c33d000-7fed1c340000 ---p 00000000 00:00 0
7fed1c340000-7fed1c43e000 rw-p 00000000 00:00 0
7fed1c43e000-7fed1c441000 ---p 00000000 00:00 0
7fed1c441000-7fed1c53f000 rw-p 00000000 00:00 0
7fed1c53f000-7fed1c542000 ---p 00000000 00:00 0
7fed1c542000-7fed1c640000 rw-p 00000000 00:00 0
7fed1c640000-7fed1c643000 ---p 00000000 00:00 0
7fed1c643000-7fed1c741000 rw-p 00000000 00:00 0
7fed1c741000-7fed1c742000 ---p 00000000 00:00 0
7fed1c742000-7fed1c842000 rw-p 00000000 00:00 0
7fed1c842000-7fed1c845000 r--s 0000f000 fd:04 131157                     /opt/app/appd_agent/ver4.0.4.0/javaagent.jar
7fed1c845000-7fed1c8b5000 rw-p 00000000 00:00 0
7fed1c8b5000-7fed1ca73000 r--s 031bf000 fd:0a 7873                       /usr/java/jre1.7.0_21/lib/rt.jar
7fed1ca73000-7fed1ca75000 r--s 0001c000 fd:0a 7452                       /usr/java/jre1.7.0_21/lib/alt-rt.jar
7fed1ca75000-7fed1cf6a000 rw-p 00000000 00:00 0
7fed1cf6a000-7fed1cf6b000 ---p 00000000 00:00 0
7fed1cf6b000-7fed2316d000 rw-p 00000000 00:00 0
7fed2316d000-7fed24edd000 rwxp 00000000 00:00 0
7fed24edd000-7fed2616d000 rw-p 00000000 00:00 0
7fed2616d000-7fed2c000000 r--p 00000000 fd:0a 24607                      /usr/lib/locale/locale-archive
7fed2c000000-7fed2cedd000 rw-p 00000000 00:00 0
7fed2cedd000-7fed30000000 ---p 00000000 00:00 0
7fed30000000-7fed30002000 r--s 00008000 fd:04 393238                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/bin/tomcat-juli.jar
7fed30002000-7fed30003000 r--s 00003000 fd:0a 7636                       /usr/java/jre1.7.0_21/lib/ext/sunec.jar
7fed30003000-7fed30007000 r--s 00036000 fd:0a 7638                       /usr/java/jre1.7.0_21/lib/ext/sunpkcs11.jar
7fed30007000-7fed3000a000 r--s 00018000 fd:0a 7680                       /usr/java/jre1.7.0_21/lib/jce.jar
7fed3000a000-7fed3001b000 r--s 000bb000 fd:04 131166                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/commons-math-2.1.jar
7fed3001b000-7fed3001d000 r--s 0000d000 fd:04 131165                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/commons-logging-1.1.1.jar
7fed3001d000-7fed30023000 r--s 00048000 fd:04 131160                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/asm-3.3.1.jar
7fed30023000-7fed30025000 r--s 00008000 fd:04 131172                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/httpmime-4.3.3.jar
7fed30025000-7fed30026000 r--s 00004000 fd:04 131171                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/httpcore-niossl-4.0-alpha6.jar
7fed30026000-7fed3002d000 r--s 0005c000 fd:04 131161                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/asm-5.0.1.jar
7fed3002d000-7fed30030000 r--s 00022000 fd:04 131168                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/httpclient-cache-4.3.3.jar
7fed30030000-7fed30036000 r--s 00033000 fd:04 131162                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/commons-codec-1.6.jar
7fed30036000-7fed3003d000 r--s 00047000 fd:04 131170                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/httpcore-nio-4.3.2.jar
7fed3003d000-7fed30040000 r--s 0001e000 fd:04 131173                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/servlet-api-2.5-6.1.3.jar
7fed30040000-7fed30049000 r--s 00050000 fd:04 131159                     /opt/app/appd_agent/ver4.0.4.0/lib/singularity-log4j.jar
7fed30049000-7fed30055000 r--s 000b0000 fd:0a 7747                       /usr/java/jre1.7.0_21/lib/resources.jar
7fed30055000-7fed300d4000 r--s 0060f000 fd:04 131158                     /opt/app/appd_agent/ver4.0.4.0/lib/appagent.jar
7fed300d4000-7fed30347000 rw-p 00000000 00:00 0
7fed30347000-7fed303b7000 rw-p 00000000 00:00 0
7fed303b7000-7fed3184b000 rw-p 00000000 00:00 0
7fed3184b000-7fed318ba000 rw-p 00000000 00:00 0
7fed318ba000-7fed318bb000 rw-p 00000000 00:00 0
7fed318bb000-7fed318bc000 ---p 00000000 00:00 0
7fed318bc000-7fed319bc000 rw-p 00000000 00:00 0
7fed319bc000-7fed319bd000 ---p 00000000 00:00 0
7fed319bd000-7fed31b33000 rw-p 00000000 00:00 0
7fed31b33000-7fed31b7d000 rw-p 00000000 00:00 0
7fed31b7d000-7fed31b97000 r-xp 00000000 fd:0a 7507                       /usr/java/jre1.7.0_21/lib/amd64/libzip.so
7fed31b97000-7fed31d97000 ---p 0001a000 fd:0a 7507                       /usr/java/jre1.7.0_21/lib/amd64/libzip.so
7fed31d97000-7fed31d98000 rw-p 0001a000 fd:0a 7507                       /usr/java/jre1.7.0_21/lib/amd64/libzip.so
7fed31d98000-7fed31da5000 r-xp 00000000 fd:01 2225                       /lib64/libnss_files-2.12.so
7fed31da5000-7fed31fa4000 ---p 0000d000 fd:01 2225                       /lib64/libnss_files-2.12.so
7fed31fa4000-7fed31fa5000 r--p 0000c000 fd:01 2225                       /lib64/libnss_files-2.12.so
7fed31fa5000-7fed31fa6000 rw-p 0000d000 fd:01 2225                       /lib64/libnss_files-2.12.so
7fed31fa6000-7fed31fa7000 r--s 00005000 fd:04 393243                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/bin/commons-daemon.jar
7fed31fa7000-7fed31fa8000 r--s 00006000 fd:04 393242                     /opt/app/installs/tomcat-7.0.39-hbcpw-live-09/bin/bootstrap.jar
7fed31fa8000-7fed31fac000 r--s 0007d000 fd:0a 7748                       /usr/java/jre1.7.0_21/lib/jsse.jar
7fed31fac000-7fed31fae000 r--s 00010000 fd:04 131176                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/zt-zip-1.8.jar
7fed31fae000-7fed31fb2000 r--s 00027000 fd:04 131164                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/commons-io-2.2.jar
7fed31fb2000-7fed31fb4000 r--s 00005000 fd:04 131175                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/slf4j-api-1.6.6.jar
7fed31fb4000-7fed31fb6000 r--s 0000f000 fd:04 131163                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/commons-fileupload-1.3.1.jar
7fed31fb6000-7fed31fc0000 r-xp 00000000 fd:0a 7472                       /usr/java/jre1.7.0_21/lib/amd64/libinstrument.so
7fed31fc0000-7fed321bf000 ---p 0000a000 fd:0a 7472                       /usr/java/jre1.7.0_21/lib/amd64/libinstrument.so
7fed321bf000-7fed321c0000 rw-p 00009000 fd:0a 7472                       /usr/java/jre1.7.0_21/lib/amd64/libinstrument.so
7fed321c0000-7fed321e9000 r-xp 00000000 fd:0a 7477                       /usr/java/jre1.7.0_21/lib/amd64/libjava.so
7fed321e9000-7fed323e9000 ---p 00029000 fd:0a 7477                       /usr/java/jre1.7.0_21/lib/amd64/libjava.so
7fed323e9000-7fed323eb000 rw-p 00029000 fd:0a 7477                       /usr/java/jre1.7.0_21/lib/amd64/libjava.so
7fed323eb000-7fed323f8000 r-xp 00000000 fd:0a 7506                       /usr/java/jre1.7.0_21/lib/amd64/libverify.so
7fed323f8000-7fed325f7000 ---p 0000d000 fd:0a 7506                       /usr/java/jre1.7.0_21/lib/amd64/libverify.so
7fed325f7000-7fed325f9000 rw-p 0000c000 fd:0a 7506                       /usr/java/jre1.7.0_21/lib/amd64/libverify.so
7fed325f9000-7fed325fc000 ---p 00000000 00:00 0
7fed325fc000-7fed326fa000 rw-p 00000000 00:00 0
7fed326fa000-7fed33144000 r-xp 00000000 fd:0a 7511                       /usr/java/jre1.7.0_21/lib/amd64/server/libjvm.so
7fed33144000-7fed33343000 ---p 00a4a000 fd:0a 7511                       /usr/java/jre1.7.0_21/lib/amd64/server/libjvm.so
7fed33343000-7fed333e7000 rw-p 00a49000 fd:0a 7511                       /usr/java/jre1.7.0_21/lib/amd64/server/libjvm.so
7fed333e7000-7fed33425000 rw-p 00000000 00:00 0
7fed33425000-7fed3343b000 r-xp 00000000 fd:0a 7460                       /usr/java/jre1.7.0_21/lib/amd64/jli/libjli.so
7fed3343b000-7fed3363b000 ---p 00016000 fd:0a 7460                       /usr/java/jre1.7.0_21/lib/amd64/jli/libjli.so
7fed3363b000-7fed3363c000 rw-p 00016000 fd:0a 7460                       /usr/java/jre1.7.0_21/lib/amd64/jli/libjli.so
7fed3363c000-7fed3363d000 rw-p 00000000 00:00 0
7fed3363d000-7fed33643000 r--s 0003f000 fd:04 131169                     /opt/app/appd_agent/ver4.0.4.0/lib/tp/httpcore-4.3.2.jar
7fed33643000-7fed3364b000 rw-s 00000000 fd:08 27                         /tmp/hsperfdata_tomcat/21352
7fed3364b000-7fed3364c000 rw-p 00000000 00:00 0
7fed3364c000-7fed3364d000 ---p 00000000 00:00 0
7fed3364d000-7fed3364e000 rw-p 00000000 00:00 0
7ffd277cc000-7ffd277e1000 rw-p 00000000 00:00 0                          [stack]
7ffd277f4000-7ffd277f5000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
jvm_args: -Djava.util.logging.config.file=/opt/app/tomcat-hbcpw-live-09/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dtomcat.name=tomcat-hbcpw-live-09 -
Xms6144m -Xmx6144m -XX:MaxPermSize=512m -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:+UseBiasedLocking -Dsun.net.inetaddr.ttl=180 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote
.port=8709 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Xmn3072M -XX:PermSize=128m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+U
seCMSInitiatingOccupancyOnly -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:+UseCompressedOops -XX:+CMSIncrementalMode -javaagent:/opt/app/appd_agent/ver4.0.4.0/javaagent.
jar -Djava.endorsed.dirs=/opt/app/tomcat-hbcpw-live-09/endorsed -Dcatalina.base=/opt/app/tomcat-hbcpw-live-09 -Dcatalina.home=/opt/app/tomcat-hbcpw-live-09 -Djava.io.tmpdir=/opt/app/tomcat-hbcpw-liv
e-09/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/app/tomcat-hbcpw-live-09/jre
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/tomcat/bin
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0x8a80e0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x8a80e0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x744170], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x744170], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x744170], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x744170], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x743ac0], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x745e50], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x745e50], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x745e50], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004


---------------  S Y S T E M  ---------------

OS:Red Hat Enterprise Linux Server release 6.3 (Santiago)

uname:Linux 2.6.32-642.6.2.el6.x86_64 #1 SMP Mon Oct 24 10:22:33 EDT 2016 x86_64
libc:glibc 2.12 NPTL 2.12
rlimit: STACK 10240k, CORE 0k, NPROC 1024, NOFILE 4000, AS infinity
load average:12.39 12.22 12.21

/proc/meminfo:
MemTotal:       14266748 kB
MemFree:          166952 kB
Buffers:          198496 kB
Cached:          4249660 kB
SwapCached:        47568 kB
Active:         10100704 kB
Inactive:        3482752 kB
Active(anon):    7698068 kB
Inactive(anon):  1437236 kB
Active(file):    2402636 kB
Inactive(file):  2045516 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      10485756 kB
SwapFree:        9702488 kB
Dirty:               228 kB
Writeback:             0 kB
AnonPages:       9124236 kB
Mapped:            75284 kB
Shmem:                 4 kB
Slab:             323596 kB
SReclaimable:     284972 kB
SUnreclaim:        38624 kB
KernelStack:       12864 kB
PageTables:        28060 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    17619128 kB
Committed_AS:   14192968 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      344296 kB
VmallocChunk:   34359353852 kB
HardwareCorrupted:     0 kB
AnonHugePages:   8873984 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       10240 kB
DirectMap2M:    14669824 kB


CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 37 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, tsc, tscinvbit

/proc/cpuinfo:
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 37
model name      : Intel(R) Xeon(R) CPU E7- 4850  @ 2.00GHz
stepping        : 1
microcode       : 54
cpu MHz         : 1994.994
cache size      : 24576 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_
reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat epb dtherm
bogomips        : 3989.98
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 37
model name      : Intel(R) Xeon(R) CPU E7- 4850  @ 2.00GHz
stepping        : 1
microcode       : 54
cpu MHz         : 1994.994
cache size      : 24576 KB
physical id     : 2
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_
reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat epb dtherm
bogomips        : 3989.98
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 14266748k(166952k free), swap 10485756k(9702488k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (23.21-b01) for linux-amd64 JRE (1.7.0_21-b11), built on Apr  4 2013 04:03:29 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Wed Jan 25 15:36:51 2017
elapsed time: 29032 seconds
\





Regards

Satish Chhatpar


::DISCLAIMER::
________________________________________________________________________________________________________________
Confidentiality Notice from Dixons Carphone plc (registered in England & Wales No.07105905) of 1 Portal Way, London, W3 6RS ("Dixons Carphone"). The information contained in this e-mail and any attachments may be legally privileged, proprietary and/or confidential. If you received this e-mail in error, please notify the sender by return, permanently delete the e-mail and destroy all hard copies immediately. No warranty is made as to the completeness or accuracy of the information contained in this e-mail. Opinions, conclusions and statements of intent in this e-mail are those of the sender and will not bind any Dixons Carphone group company (Dixons Carphone Group) unless confirmed by an authorised representative independently of this e-mail. We do not accept responsibility for viruses; you must scan for these. E-mails sent to and from Dixons Carphone Group are routinely monitored for record keeping, quality control, training purposes, to ensure regulatory compliance and to prevent viruses and unauthorised use of our computer systems. The Carphone Warehouse Limited (registered in England & Wales No.02142673) is a member of the Dixons Carphone Group and is authorised and regulated by the Financial Conduct Authority.
________________________________________________________________________________________________________________

Re: Apache Tomcat/7.0.39 crashed with fatal error

Posted by Coty Sutherland <cs...@redhat.com>.
> # Problematic frame:
> # J  org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List;

Generally a crash in a java frame is a JVM bug. A quick google search
of the problematic frame yields the following first result,
https://github.com/rholder/jvm-loop-unswitching-bug. Looking at the
java7 bug linked on the page
(http://bugs.java.com/view_bug.do?bug_id=8025398) and further details
from the bug that the fix was backported from
(http://bugs.java.com/view_bug.do?bug_id=8021898) shows evidence that
this is likely your issue. You'll need to update your JDK or use the
suggested workaround to prevent future occurrences.

On Thu, Jan 26, 2017 at 4:40 PM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Aurélien,
>
> On 1/26/17 4:31 PM, Aurélien Terrestris wrote:
>> maybe you're just sending cookies with non-compliant characters.
>> Please check what you're sending if you can reproduce this problem
>> yourself
>>
>> RFC 6265 says  :
>>
>> cookie-value      = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
>> cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E ;
>> US-ASCII characters excluding CTLs, ; whitespace DQUOTE, comma,
>> semicolon, ; and backslash
>
> Even if the client is sending a malformed HTTP header (or cookie,
> specifically), it shouldn't crash the JVM.
>
> - -chris
>
>> 2017-01-26 22:22 GMT+01:00 Satish Chhatpar 02
>> <Ch...@cpwplc.com>:
>>
>>> Yes all of them failed in the same way.
>>>
>>>
>>> # Problematic frame: # J
>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(
>>> Ljava/util/List;)Ljava/util/List;
>>>
>>>
>>>
>>> Regards
>>>
>>> Satish Chhatpar
>>>
>>>
>>> ________________________________ From: Christopher Schultz
>>> <ch...@christopherschultz.net> Sent: Friday, January 27, 2017
>>> 2:44:54 AM To: Tomcat Users List Subject: Re: Apache
>>> Tomcat/7.0.39 crashed with fatal error
>>>
>> Satish,
>>
>> On 1/26/17 3:42 PM, Satish Chhatpar 02 wrote:
>>>>> Thanks Chris. I appreciate your help.
>>>>>
>>>>> All 4 tomcats are on diff machines. One on each, with same
>>>>> tomcat version, same java version and same OS for all.
>>
>> Did they all fail in the same way (JVM crash @
>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies)?
>>
>>>>> Tomcats are not in cluster.
>>
>> I would highly recommend upgrading the JVM on one of those servers
>> to 1.7.latest to see if everything still works. If things go well,
>>  upgrade all of them.
>>
>> Then deploy the 1.8.latest to one of them. Tomcat shouldn't have
>> any compatibility issues with Java 8, but you will definitely want
>> to test everything in your application of course.
>>
>> -chris
>>
>>>>> ________________________________ From: Christopher Schultz
>>>>> <ch...@christopherschultz.net> Sent: Friday, January 27, 2017
>>>>> 1:52:47 AM To: Tomcat Users List Subject: Re: Apache
>>>>> Tomcat/7.0.39 crashed with fatal error
>>>>>
>>>>> Satish,
>>>>>
>>>>> On 1/26/17 2:28 PM, Satish Chhatpar 02 wrote:
>>>>>> we are using Apache Tomcat/7.0.39 for our java
>>>>>> application.
>>>>>
>>>>> I highly recommend an upgrade for both Tomcat and Java.
>>>>> There are published vulnerabilities for both product versions
>>>>> you are using.
>>>>>
>>>>>> There are 4 tomcat instances using same tomcat version and
>>>>>> java version. yesterday all 4 tomcats crashed with below
>>>>>> error in hs_err_pid log file.
>>>>>
>>>>> All on the same hardware? Or separate machines?
>>>>>
>>>>>> This log file was created for all 4 tomcats.
>>>>>
>>>>>> Its very peculiar behaviour that all 4 crashed around same
>>>>>> time.
>>>>>
>>>>> If they are in a cluster, one going down could cause the
>>>>> load on the others to go up, increasing the chances of a
>>>>> problem.
>>>>>
>>>>>> Any information can help us to mitigate this incident.
>>>>>
>>>>>> Apache Tomcat/7.0.39
>>>>>
>>>>> Unless this is a package-managed version of Tomcat with an
>>>>> unfortunately inaccurate version number, that version of
>>>>> Tomcat is nearly 3 years old. The current version in the
>>>>> 7.0.x line is 7.0.75 (released yesterday).
>>>>>
>>>>>> java version "1.7.0_21" Java(TM) SE Runtime Environment
>>>>>> (build 1.7.0_21-b11) Java HotSpot(TM) 64-Bit Server VM
>>>>>> (build 23.21-b01, mixed mode)
>>>>>
>>>>> That version of Java is also nearly 3 years old. Latest 1.7
>>>>> build is 1.7.0_80 release nearly 3 years ago. Note that Java
>>>>> 7 is no longer supported unless you have a long-term support
>>>>> contract with Oracle, in which case the latest version is
>>>>> 1.7.0_131, released earlier this month.
>>>>>
>>>>>> OS used
>>>>>
>>>>>
>>>>>> Red Hat Enterprise Linux Server release 6.3 (Santiago)
>>>>>
>>>>> Ouch! 5 years old!
>>>>>
>>>>>> # # A fatal error has been detected by the Java Runtime
>>>>>> Environment: # #  SIGSEGV (0xb) at pc=0x00007fed24ecfe9e,
>>>>>> pid=21352, tid=140656275650304 # # JRE version: 7.0_21-b11
>>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01
>>>>>> mixed mode linux-amd64 compressed oops) # Problematic
>>>>>> frame: # J
>>>>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util
> /Li
>>
>>>>>>
>>>>>>
> s
>>>>>
>>>>>>
>> t;)Ljava/util/List;
>>>>>
>>>>>
>>>>> #
>>>>>> # Failed to write core dump. Core dumps have been
>>>>>> disabled. To enable core dumping, try "ulimit -c unlimited"
>>>>>> before starting Java again # # If you would like to submit
>>>>>> a bug report, please visit: #
>>>>>> http://bugreport.sun.com/bugreport/crash.jsp #
>>>>>
>>>>> This is either a JVM bug is a hardware error. Given that the
>>>>> OS if 5 years old, I'm guessing the hardware is at least
>>>>> that old. I'd expect 5-year old hardware to be fairly
>>>>> trustworthy, but it may not have been properly-tested before
>>>>> going into production.
>>>>>
>>>>> If it's all on a single piece of hardware (all 4 Tomcats),
>>>>> I'd blame the hardware and look for a speedy replacement
>>>>> (properly-tested this time). If it was on different
>>>>> machines, I'd suspect a JVM bug.
>>>>>
>>>>> -chris
>>>>>
>>>>> -------------------------------------------------------------------
> - --
>>>>>
>>>>>
>>
>>>>>
>>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail:
>>>>> users-help@tomcat.apache.org
>>>>>
>>>>> ::DISCLAIMER::
>>>>> ___________________________________________________________________
> ___
>>
>>>>>
>>>>>
> __________________________________________
>>>>>
>>>>>
>> Confidentiality Notice from Dixons Carphone plc (registered in
>> England & Wales No.07105905) of 1 Portal Way, London, W3 6RS
>> ("Dixons Carphone"). The information contained in this e-mail and
>> any attachments may be legally privileged, proprietary and/or
>> confidential. If you received this e-mail in error, please notify
>> the sender by return, permanently delete the e-mail and destroy
>> all hard copies immediately. No warranty is made as to the
>> completeness or accuracy of the information contained in this
>> e-mail. Opinions, conclusions and statements of intent in this
>> e-mail are those of the sender and will not bind any Dixons
>> Carphone group company (Dixons Carphone Group) unless confirmed by
>> an authorised representative independently of this e-mail. We do
>> not accept responsibility for viruses; you must scan for these.
>> E-mails sent to and from Dixons Carphone Group are routinely
>> monitored for record keeping, quality control, training purposes,
>> to ensure regulatory compliance and to prevent viruses and
>> unauthorised use of our computer systems. The Carphone Warehouse
>> Limited (registered in England & Wales No.02142673) is a member of
>> the Dixons Carphone Group and is authorised and regulated by the
>> Financial Conduct Authority.
>>>>> ___________________________________________________________________
> ___
>>
>>>>>
>>>>>
> __________________________________________
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>>
>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJYimzGAAoJEBzwKT+lPKRYQMUQAK7yrP+4eL2vG24fs5OG4qf+
> O/Vy1Arl5QVvxm8avjB7458W2fgkFP/f15wn/jZdsMAF1VL7C+EZ4t5QfcBf2oUE
> kawWmPU4qMvCG5mIeiYqRHVpVUyJ8PSX+jxOqJR039MQ4Kgr/AiaRwqZ/lKrN+ye
> c/jH3ZF3mVZcJAQLdUc7ABQjz3w0MDGJFGhz+aptQ4Zmny2/vyQTlnLdm9+JlCc+
> XmDqu8V6Lz1QX23UopFafv17bEmqXkLvpRHfiaPtzHDeX9ZciigeK9TIK279CnjS
> lcJQPJGa4XH+u7ItNk1Wvv7l/YC+G159gg2RzauOe7x7QzTHiY5CSd9klRoW4H0p
> wzNXmrUHoFK9BW6lCfpd9LKVADc6ggCXkBfQHlca/sNhwgRHeA90tNEIEBvLoMEb
> WqN7igF/+7aFctaSBtr1KCbx/R9FL8tqUwCbS4gIdbwQ7FnEx9DTHug5wMhgp9GV
> SWLVb/NLYH8YJEEXMBROroyY+82UgK7ynYaYTzvR7P1zmnD9jeOTI792crlDWNzv
> /zP1x0iRX7zJ33H7pEG6xdw/SHR+dFKnEu3bxn5QawupOxQt9ifWXxUgTXx/+Q/m
> 7FmRwZDpfxDngsdVgKdQOlOxp2PGHRlu7rVkq4LzXXsHsUZsVoEtVEKvqKarTd45
> z13tY8pReCBVV3nRdkAU
> =SOql
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Apache Tomcat/7.0.39 crashed with fatal error

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Aur�lien,

On 1/26/17 4:31 PM, Aur�lien Terrestris wrote:
> maybe you're just sending cookies with non-compliant characters. 
> Please check what you're sending if you can reproduce this problem 
> yourself
> 
> RFC 6265 says  :
> 
> cookie-value      = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
> cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E ;
> US-ASCII characters excluding CTLs, ; whitespace DQUOTE, comma, 
> semicolon, ; and backslash

Even if the client is sending a malformed HTTP header (or cookie,
specifically), it shouldn't crash the JVM.

- -chris

> 2017-01-26 22:22 GMT+01:00 Satish Chhatpar 02 
> <Ch...@cpwplc.com>:
> 
>> Yes all of them failed in the same way.
>> 
>> 
>> # Problematic frame: # J 
>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies( 
>> Ljava/util/List;)Ljava/util/List;
>> 
>> 
>> 
>> Regards
>> 
>> Satish Chhatpar
>> 
>> 
>> ________________________________ From: Christopher Schultz 
>> <ch...@christopherschultz.net> Sent: Friday, January 27, 2017 
>> 2:44:54 AM To: Tomcat Users List Subject: Re: Apache 
>> Tomcat/7.0.39 crashed with fatal error
>> 
> Satish,
> 
> On 1/26/17 3:42 PM, Satish Chhatpar 02 wrote:
>>>> Thanks Chris. I appreciate your help.
>>>> 
>>>> All 4 tomcats are on diff machines. One on each, with same 
>>>> tomcat version, same java version and same OS for all.
> 
> Did they all fail in the same way (JVM crash @ 
> org.apache.http.impl.cookie.BestMatchSpec.formatCookies)?
> 
>>>> Tomcats are not in cluster.
> 
> I would highly recommend upgrading the JVM on one of those servers 
> to 1.7.latest to see if everything still works. If things go well,
>  upgrade all of them.
> 
> Then deploy the 1.8.latest to one of them. Tomcat shouldn't have 
> any compatibility issues with Java 8, but you will definitely want 
> to test everything in your application of course.
> 
> -chris
> 
>>>> ________________________________ From: Christopher Schultz 
>>>> <ch...@christopherschultz.net> Sent: Friday, January 27, 2017
>>>> 1:52:47 AM To: Tomcat Users List Subject: Re: Apache 
>>>> Tomcat/7.0.39 crashed with fatal error
>>>> 
>>>> Satish,
>>>> 
>>>> On 1/26/17 2:28 PM, Satish Chhatpar 02 wrote:
>>>>> we are using Apache Tomcat/7.0.39 for our java 
>>>>> application.
>>>> 
>>>> I highly recommend an upgrade for both Tomcat and Java.
>>>> There are published vulnerabilities for both product versions
>>>> you are using.
>>>> 
>>>>> There are 4 tomcat instances using same tomcat version and 
>>>>> java version. yesterday all 4 tomcats crashed with below 
>>>>> error in hs_err_pid log file.
>>>> 
>>>> All on the same hardware? Or separate machines?
>>>> 
>>>>> This log file was created for all 4 tomcats.
>>>> 
>>>>> Its very peculiar behaviour that all 4 crashed around same 
>>>>> time.
>>>> 
>>>> If they are in a cluster, one going down could cause the
>>>> load on the others to go up, increasing the chances of a
>>>> problem.
>>>> 
>>>>> Any information can help us to mitigate this incident.
>>>> 
>>>>> Apache Tomcat/7.0.39
>>>> 
>>>> Unless this is a package-managed version of Tomcat with an 
>>>> unfortunately inaccurate version number, that version of 
>>>> Tomcat is nearly 3 years old. The current version in the 
>>>> 7.0.x line is 7.0.75 (released yesterday).
>>>> 
>>>>> java version "1.7.0_21" Java(TM) SE Runtime Environment 
>>>>> (build 1.7.0_21-b11) Java HotSpot(TM) 64-Bit Server VM 
>>>>> (build 23.21-b01, mixed mode)
>>>> 
>>>> That version of Java is also nearly 3 years old. Latest 1.7 
>>>> build is 1.7.0_80 release nearly 3 years ago. Note that Java 
>>>> 7 is no longer supported unless you have a long-term support 
>>>> contract with Oracle, in which case the latest version is 
>>>> 1.7.0_131, released earlier this month.
>>>> 
>>>>> OS used
>>>> 
>>>> 
>>>>> Red Hat Enterprise Linux Server release 6.3 (Santiago)
>>>> 
>>>> Ouch! 5 years old!
>>>> 
>>>>> # # A fatal error has been detected by the Java Runtime 
>>>>> Environment: # #  SIGSEGV (0xb) at pc=0x00007fed24ecfe9e, 
>>>>> pid=21352, tid=140656275650304 # # JRE version: 7.0_21-b11 
>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 
>>>>> mixed mode linux-amd64 compressed oops) # Problematic 
>>>>> frame: # J 
>>>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util
/Li
>
>>>>>
>>>>> 
s
>>>> 
>>>>> 
> t;)Ljava/util/List;
>>>> 
>>>> 
>>>> #
>>>>> # Failed to write core dump. Core dumps have been
>>>>> disabled. To enable core dumping, try "ulimit -c unlimited"
>>>>> before starting Java again # # If you would like to submit
>>>>> a bug report, please visit: # 
>>>>> http://bugreport.sun.com/bugreport/crash.jsp #
>>>> 
>>>> This is either a JVM bug is a hardware error. Given that the 
>>>> OS if 5 years old, I'm guessing the hardware is at least
>>>> that old. I'd expect 5-year old hardware to be fairly
>>>> trustworthy, but it may not have been properly-tested before
>>>> going into production.
>>>> 
>>>> If it's all on a single piece of hardware (all 4 Tomcats), 
>>>> I'd blame the hardware and look for a speedy replacement 
>>>> (properly-tested this time). If it was on different
>>>> machines, I'd suspect a JVM bug.
>>>> 
>>>> -chris
>>>> 
>>>> -------------------------------------------------------------------
- --
>>>>
>>>>
>
>>>>
>>>> 
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: 
>>>> users-help@tomcat.apache.org
>>>> 
>>>> ::DISCLAIMER:: 
>>>> ___________________________________________________________________
___
>
>>>>
>>>> 
__________________________________________
>>>> 
>>>> 
> Confidentiality Notice from Dixons Carphone plc (registered in 
> England & Wales No.07105905) of 1 Portal Way, London, W3 6RS 
> ("Dixons Carphone"). The information contained in this e-mail and 
> any attachments may be legally privileged, proprietary and/or 
> confidential. If you received this e-mail in error, please notify 
> the sender by return, permanently delete the e-mail and destroy
> all hard copies immediately. No warranty is made as to the
> completeness or accuracy of the information contained in this
> e-mail. Opinions, conclusions and statements of intent in this
> e-mail are those of the sender and will not bind any Dixons
> Carphone group company (Dixons Carphone Group) unless confirmed by
> an authorised representative independently of this e-mail. We do
> not accept responsibility for viruses; you must scan for these.
> E-mails sent to and from Dixons Carphone Group are routinely
> monitored for record keeping, quality control, training purposes,
> to ensure regulatory compliance and to prevent viruses and
> unauthorised use of our computer systems. The Carphone Warehouse
> Limited (registered in England & Wales No.02142673) is a member of
> the Dixons Carphone Group and is authorised and regulated by the
> Financial Conduct Authority.
>>>> ___________________________________________________________________
___
>
>>>>
>>>> 
__________________________________________
>>>> 
>> 
>> ---------------------------------------------------------------------
>>
>>
>> 
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>> 
>> 
> 
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYimzGAAoJEBzwKT+lPKRYQMUQAK7yrP+4eL2vG24fs5OG4qf+
O/Vy1Arl5QVvxm8avjB7458W2fgkFP/f15wn/jZdsMAF1VL7C+EZ4t5QfcBf2oUE
kawWmPU4qMvCG5mIeiYqRHVpVUyJ8PSX+jxOqJR039MQ4Kgr/AiaRwqZ/lKrN+ye
c/jH3ZF3mVZcJAQLdUc7ABQjz3w0MDGJFGhz+aptQ4Zmny2/vyQTlnLdm9+JlCc+
XmDqu8V6Lz1QX23UopFafv17bEmqXkLvpRHfiaPtzHDeX9ZciigeK9TIK279CnjS
lcJQPJGa4XH+u7ItNk1Wvv7l/YC+G159gg2RzauOe7x7QzTHiY5CSd9klRoW4H0p
wzNXmrUHoFK9BW6lCfpd9LKVADc6ggCXkBfQHlca/sNhwgRHeA90tNEIEBvLoMEb
WqN7igF/+7aFctaSBtr1KCbx/R9FL8tqUwCbS4gIdbwQ7FnEx9DTHug5wMhgp9GV
SWLVb/NLYH8YJEEXMBROroyY+82UgK7ynYaYTzvR7P1zmnD9jeOTI792crlDWNzv
/zP1x0iRX7zJ33H7pEG6xdw/SHR+dFKnEu3bxn5QawupOxQt9ifWXxUgTXx/+Q/m
7FmRwZDpfxDngsdVgKdQOlOxp2PGHRlu7rVkq4LzXXsHsUZsVoEtVEKvqKarTd45
z13tY8pReCBVV3nRdkAU
=SOql
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Apache Tomcat/7.0.39 crashed with fatal error

Posted by Aurélien Terrestris <at...@gmail.com>.
Hello

maybe you're just sending cookies with non-compliant characters. Please
check what you're sending if you can reproduce this problem yourself

RFC 6265 says  :

 cookie-value      = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
 cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
                       ; US-ASCII characters excluding CTLs,
                       ; whitespace DQUOTE, comma, semicolon,
                       ; and backslash




2017-01-26 22:22 GMT+01:00 Satish Chhatpar 02 <Ch...@cpwplc.com>:

> Yes all of them failed in the same way.
>
>
> # Problematic frame:
> # J  org.apache.http.impl.cookie.BestMatchSpec.formatCookies(
> Ljava/util/List;)Ljava/util/List;
>
>
>
> Regards
>
> Satish Chhatpar
>
>
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Friday, January 27, 2017 2:44:54 AM
> To: Tomcat Users List
> Subject: Re: Apache Tomcat/7.0.39 crashed with fatal error
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Satish,
>
> On 1/26/17 3:42 PM, Satish Chhatpar 02 wrote:
> > Thanks Chris. I appreciate your help.
> >
> > All 4 tomcats are on diff machines. One on each, with same tomcat
> > version, same java version and same OS for all.
>
> Did they all fail in the same way (JVM crash @
> org.apache.http.impl.cookie.BestMatchSpec.formatCookies)?
>
> > Tomcats are not in cluster.
>
> I would highly recommend upgrading the JVM on one of those servers to
> 1.7.latest to see if everything still works. If things go well,
> upgrade all of them.
>
> Then deploy the 1.8.latest to one of them. Tomcat shouldn't have any
> compatibility issues with Java 8, but you will definitely want to test
> everything in your application of course.
>
> - -chris
>
> > ________________________________ From: Christopher Schultz
> > <ch...@christopherschultz.net> Sent: Friday, January 27, 2017
> > 1:52:47 AM To: Tomcat Users List Subject: Re: Apache Tomcat/7.0.39
> > crashed with fatal error
> >
> > Satish,
> >
> > On 1/26/17 2:28 PM, Satish Chhatpar 02 wrote:
> >> we are using Apache Tomcat/7.0.39 for our java application.
> >
> > I highly recommend an upgrade for both Tomcat and Java. There are
> > published vulnerabilities for both product versions you are using.
> >
> >> There are 4 tomcat instances using same tomcat version and java
> >> version. yesterday all 4 tomcats crashed with below error in
> >> hs_err_pid log file.
> >
> > All on the same hardware? Or separate machines?
> >
> >> This log file was created for all 4 tomcats.
> >
> >> Its very peculiar behaviour that all 4 crashed around same time.
> >
> > If they are in a cluster, one going down could cause the load on
> > the others to go up, increasing the chances of a problem.
> >
> >> Any information can help us to mitigate this incident.
> >
> >> Apache Tomcat/7.0.39
> >
> > Unless this is a package-managed version of Tomcat with an
> > unfortunately inaccurate version number, that version of Tomcat is
> > nearly 3 years old. The current version in the 7.0.x line is
> > 7.0.75 (released yesterday).
> >
> >> java version "1.7.0_21" Java(TM) SE Runtime Environment (build
> >> 1.7.0_21-b11) Java HotSpot(TM) 64-Bit Server VM (build
> >> 23.21-b01, mixed mode)
> >
> > That version of Java is also nearly 3 years old. Latest 1.7 build
> > is 1.7.0_80 release nearly 3 years ago. Note that Java 7 is no
> > longer supported unless you have a long-term support contract with
> > Oracle, in which case the latest version is 1.7.0_131, released
> > earlier this month.
> >
> >> OS used
> >
> >
> >> Red Hat Enterprise Linux Server release 6.3 (Santiago)
> >
> > Ouch! 5 years old!
> >
> >> # # A fatal error has been detected by the Java Runtime
> >> Environment: # #  SIGSEGV (0xb) at pc=0x00007fed24ecfe9e,
> >> pid=21352, tid=140656275650304 # # JRE version: 7.0_21-b11 #
> >> Java VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode
> >> linux-amd64 compressed oops) # Problematic frame: # J
> >> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/Li
> s
> >
> >>
> t;)Ljava/util/List;
> >
> >
> > #
> >> # Failed to write core dump. Core dumps have been disabled. To
> >> enable core dumping, try "ulimit -c unlimited" before starting
> >> Java again # # If you would like to submit a bug report, please
> >> visit: # http://bugreport.sun.com/bugreport/crash.jsp #
> >
> > This is either a JVM bug is a hardware error. Given that the OS if
> > 5 years old, I'm guessing the hardware is at least that old. I'd
> > expect 5-year old hardware to be fairly trustworthy, but it may not
> > have been properly-tested before going into production.
> >
> > If it's all on a single piece of hardware (all 4 Tomcats), I'd
> > blame the hardware and look for a speedy replacement
> > (properly-tested this time). If it was on different machines, I'd
> > suspect a JVM bug.
> >
> > -chris
> >
> > ---------------------------------------------------------------------
> >
> >
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> > ::DISCLAIMER::
> > ______________________________________________________________________
> __________________________________________
> >
> >
> Confidentiality Notice from Dixons Carphone plc (registered in England &
> Wales No.07105905) of 1 Portal Way, London, W3 6RS ("Dixons Carphone").
> The information contained in this e-mail and any attachments may be
> legally privileged, proprietary and/or confidential. If you received
> this e-mail in error, please notify the sender by return, permanently
> delete the e-mail and destroy all hard copies immediately. No warranty
> is made as to the completeness or accuracy of the information contained
> in this e-mail. Opinions, conclusions and statements of intent in this
> e-mail are those of the sender and will not bind any Dixons Carphone
> group company (Dixons Carphone Group) unless confirmed by an authorised
> representative independently of this e-mail. We do not accept
> responsibility for viruses; you must scan for these. E-mails sent to and
> from Dixons Carphone Group are routinely monitored for record keeping,
> quality control, training purposes, to ensure regulatory compliance and
> to prevent viruses and unauthorised use of our computer systems. The
> Carphone Warehouse Limited (registered in England & Wales No.02142673)
> is a member of the Dixons Carphone Group and is authorised and regulated
> by the Financial Conduct Authority.
> > ______________________________________________________________________
> __________________________________________
> >
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJYimbOAAoJEBzwKT+lPKRYm3EQAL8Vgr7j8o1IPWtFX9U2v3Ce
> lXgf6b3mLP+DheBMKKidhSArQGzTa0GmYxwQ0NL4Cx0UYrrlXOsVOJWtzpRZpDKh
> Cj0MR9L3pc/2tSiKDCr4f+gxnIfRKrHCJPjU993EQO3WbsqebD85oWEESjJBskVB
> tTzbVQTHSvJYbO7rWigH1gTUPM18AqdgqwW2msnvTeUMZ142OIuvt7FzSLI9sW8E
> hdQLRyi1u9X4kZUO6Ob+q6cbEt81TLE7goJCNZ7B+PJoxRFN7H4TPlPaYIX9sw20
> oOCvySK8dH1BQJ+ufOX+lr8pDpu7vg/PmPCfFcTP705HJioc5G4yI61TIALMq2Qe
> a4FImNYbwm07SKQbTSj16PZZ7mKQZqs4UcqdrwJ5qazZXqhTpwgF/qfeu8c1d8d3
> BuwTkLJa6BM5n5VU37rCucOE4twM/9nFTnAgdB4kbfHxa6pZzXF9sFvTmPvCLBci
> CS/sOcEGbsg1kkHi1iDVFOnhafbG9hGg/alJsnWNJc7XdGXwiL/0Dk64/5HKqAs6
> LT/4AYhG/+bwsuEbIKP2zOt2d9fl7NkaI8LJLLgVoAx/JFcLa3+5K4SoM3xycM/Y
> aq6TEsMM/IsKoApTkTBWzUEK0lCNPC7MzzBdQQ3bZqmm5VYb2zwCU9TgPi6GnxMx
> 66GWAL1NlOVQuffsl8D/
> =Wkrw
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: Apache Tomcat/7.0.39 crashed with fatal error

Posted by Satish Chhatpar 02 <Ch...@cpwplc.com>.
Yes all of them failed in the same way.


# Problematic frame:
# J  org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List;



Regards

Satish Chhatpar


________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Friday, January 27, 2017 2:44:54 AM
To: Tomcat Users List
Subject: Re: Apache Tomcat/7.0.39 crashed with fatal error

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Satish,

On 1/26/17 3:42 PM, Satish Chhatpar 02 wrote:
> Thanks Chris. I appreciate your help.
>
> All 4 tomcats are on diff machines. One on each, with same tomcat
> version, same java version and same OS for all.

Did they all fail in the same way (JVM crash @
org.apache.http.impl.cookie.BestMatchSpec.formatCookies)?

> Tomcats are not in cluster.

I would highly recommend upgrading the JVM on one of those servers to
1.7.latest to see if everything still works. If things go well,
upgrade all of them.

Then deploy the 1.8.latest to one of them. Tomcat shouldn't have any
compatibility issues with Java 8, but you will definitely want to test
everything in your application of course.

- -chris

> ________________________________ From: Christopher Schultz
> <ch...@christopherschultz.net> Sent: Friday, January 27, 2017
> 1:52:47 AM To: Tomcat Users List Subject: Re: Apache Tomcat/7.0.39
> crashed with fatal error
>
> Satish,
>
> On 1/26/17 2:28 PM, Satish Chhatpar 02 wrote:
>> we are using Apache Tomcat/7.0.39 for our java application.
>
> I highly recommend an upgrade for both Tomcat and Java. There are
> published vulnerabilities for both product versions you are using.
>
>> There are 4 tomcat instances using same tomcat version and java
>> version. yesterday all 4 tomcats crashed with below error in
>> hs_err_pid log file.
>
> All on the same hardware? Or separate machines?
>
>> This log file was created for all 4 tomcats.
>
>> Its very peculiar behaviour that all 4 crashed around same time.
>
> If they are in a cluster, one going down could cause the load on
> the others to go up, increasing the chances of a problem.
>
>> Any information can help us to mitigate this incident.
>
>> Apache Tomcat/7.0.39
>
> Unless this is a package-managed version of Tomcat with an
> unfortunately inaccurate version number, that version of Tomcat is
> nearly 3 years old. The current version in the 7.0.x line is
> 7.0.75 (released yesterday).
>
>> java version "1.7.0_21" Java(TM) SE Runtime Environment (build
>> 1.7.0_21-b11) Java HotSpot(TM) 64-Bit Server VM (build
>> 23.21-b01, mixed mode)
>
> That version of Java is also nearly 3 years old. Latest 1.7 build
> is 1.7.0_80 release nearly 3 years ago. Note that Java 7 is no
> longer supported unless you have a long-term support contract with
> Oracle, in which case the latest version is 1.7.0_131, released
> earlier this month.
>
>> OS used
>
>
>> Red Hat Enterprise Linux Server release 6.3 (Santiago)
>
> Ouch! 5 years old!
>
>> # # A fatal error has been detected by the Java Runtime
>> Environment: # #  SIGSEGV (0xb) at pc=0x00007fed24ecfe9e,
>> pid=21352, tid=140656275650304 # # JRE version: 7.0_21-b11 #
>> Java VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode
>> linux-amd64 compressed oops) # Problematic frame: # J
>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/Li
s
>
>>
t;)Ljava/util/List;
>
>
> #
>> # Failed to write core dump. Core dumps have been disabled. To
>> enable core dumping, try "ulimit -c unlimited" before starting
>> Java again # # If you would like to submit a bug report, please
>> visit: # http://bugreport.sun.com/bugreport/crash.jsp #
>
> This is either a JVM bug is a hardware error. Given that the OS if
> 5 years old, I'm guessing the hardware is at least that old. I'd
> expect 5-year old hardware to be fairly trustworthy, but it may not
> have been properly-tested before going into production.
>
> If it's all on a single piece of hardware (all 4 Tomcats), I'd
> blame the hardware and look for a speedy replacement
> (properly-tested this time). If it was on different machines, I'd
> suspect a JVM bug.
>
> -chris
>
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> ::DISCLAIMER::
> ______________________________________________________________________
__________________________________________
>
>
Confidentiality Notice from Dixons Carphone plc (registered in England &
Wales No.07105905) of 1 Portal Way, London, W3 6RS ("Dixons Carphone").
The information contained in this e-mail and any attachments may be
legally privileged, proprietary and/or confidential. If you received
this e-mail in error, please notify the sender by return, permanently
delete the e-mail and destroy all hard copies immediately. No warranty
is made as to the completeness or accuracy of the information contained
in this e-mail. Opinions, conclusions and statements of intent in this
e-mail are those of the sender and will not bind any Dixons Carphone
group company (Dixons Carphone Group) unless confirmed by an authorised
representative independently of this e-mail. We do not accept
responsibility for viruses; you must scan for these. E-mails sent to and
from Dixons Carphone Group are routinely monitored for record keeping,
quality control, training purposes, to ensure regulatory compliance and
to prevent viruses and unauthorised use of our computer systems. The
Carphone Warehouse Limited (registered in England & Wales No.02142673)
is a member of the Dixons Carphone Group and is authorised and regulated
by the Financial Conduct Authority.
> ______________________________________________________________________
__________________________________________
>
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYimbOAAoJEBzwKT+lPKRYm3EQAL8Vgr7j8o1IPWtFX9U2v3Ce
lXgf6b3mLP+DheBMKKidhSArQGzTa0GmYxwQ0NL4Cx0UYrrlXOsVOJWtzpRZpDKh
Cj0MR9L3pc/2tSiKDCr4f+gxnIfRKrHCJPjU993EQO3WbsqebD85oWEESjJBskVB
tTzbVQTHSvJYbO7rWigH1gTUPM18AqdgqwW2msnvTeUMZ142OIuvt7FzSLI9sW8E
hdQLRyi1u9X4kZUO6Ob+q6cbEt81TLE7goJCNZ7B+PJoxRFN7H4TPlPaYIX9sw20
oOCvySK8dH1BQJ+ufOX+lr8pDpu7vg/PmPCfFcTP705HJioc5G4yI61TIALMq2Qe
a4FImNYbwm07SKQbTSj16PZZ7mKQZqs4UcqdrwJ5qazZXqhTpwgF/qfeu8c1d8d3
BuwTkLJa6BM5n5VU37rCucOE4twM/9nFTnAgdB4kbfHxa6pZzXF9sFvTmPvCLBci
CS/sOcEGbsg1kkHi1iDVFOnhafbG9hGg/alJsnWNJc7XdGXwiL/0Dk64/5HKqAs6
LT/4AYhG/+bwsuEbIKP2zOt2d9fl7NkaI8LJLLgVoAx/JFcLa3+5K4SoM3xycM/Y
aq6TEsMM/IsKoApTkTBWzUEK0lCNPC7MzzBdQQ3bZqmm5VYb2zwCU9TgPi6GnxMx
66GWAL1NlOVQuffsl8D/
=Wkrw
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Apache Tomcat/7.0.39 crashed with fatal error

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Satish,

On 1/26/17 3:42 PM, Satish Chhatpar 02 wrote:
> Thanks Chris. I appreciate your help.
> 
> All 4 tomcats are on diff machines. One on each, with same tomcat 
> version, same java version and same OS for all.

Did they all fail in the same way (JVM crash @
org.apache.http.impl.cookie.BestMatchSpec.formatCookies)?

> Tomcats are not in cluster.

I would highly recommend upgrading the JVM on one of those servers to
1.7.latest to see if everything still works. If things go well,
upgrade all of them.

Then deploy the 1.8.latest to one of them. Tomcat shouldn't have any
compatibility issues with Java 8, but you will definitely want to test
everything in your application of course.

- -chris

> ________________________________ From: Christopher Schultz
> <ch...@christopherschultz.net> Sent: Friday, January 27, 2017
> 1:52:47 AM To: Tomcat Users List Subject: Re: Apache Tomcat/7.0.39
> crashed with fatal error
> 
> Satish,
> 
> On 1/26/17 2:28 PM, Satish Chhatpar 02 wrote:
>> we are using Apache Tomcat/7.0.39 for our java application.
> 
> I highly recommend an upgrade for both Tomcat and Java. There are 
> published vulnerabilities for both product versions you are using.
> 
>> There are 4 tomcat instances using same tomcat version and java 
>> version. yesterday all 4 tomcats crashed with below error in 
>> hs_err_pid log file.
> 
> All on the same hardware? Or separate machines?
> 
>> This log file was created for all 4 tomcats.
> 
>> Its very peculiar behaviour that all 4 crashed around same time.
> 
> If they are in a cluster, one going down could cause the load on
> the others to go up, increasing the chances of a problem.
> 
>> Any information can help us to mitigate this incident.
> 
>> Apache Tomcat/7.0.39
> 
> Unless this is a package-managed version of Tomcat with an 
> unfortunately inaccurate version number, that version of Tomcat is 
> nearly 3 years old. The current version in the 7.0.x line is
> 7.0.75 (released yesterday).
> 
>> java version "1.7.0_21" Java(TM) SE Runtime Environment (build 
>> 1.7.0_21-b11) Java HotSpot(TM) 64-Bit Server VM (build
>> 23.21-b01, mixed mode)
> 
> That version of Java is also nearly 3 years old. Latest 1.7 build
> is 1.7.0_80 release nearly 3 years ago. Note that Java 7 is no
> longer supported unless you have a long-term support contract with
> Oracle, in which case the latest version is 1.7.0_131, released
> earlier this month.
> 
>> OS used
> 
> 
>> Red Hat Enterprise Linux Server release 6.3 (Santiago)
> 
> Ouch! 5 years old!
> 
>> # # A fatal error has been detected by the Java Runtime 
>> Environment: # #  SIGSEGV (0xb) at pc=0x00007fed24ecfe9e, 
>> pid=21352, tid=140656275650304 # # JRE version: 7.0_21-b11 #
>> Java VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode 
>> linux-amd64 compressed oops) # Problematic frame: # J 
>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/Li
s
>
>> 
t;)Ljava/util/List;
> 
> 
> #
>> # Failed to write core dump. Core dumps have been disabled. To 
>> enable core dumping, try "ulimit -c unlimited" before starting
>> Java again # # If you would like to submit a bug report, please
>> visit: # http://bugreport.sun.com/bugreport/crash.jsp #
> 
> This is either a JVM bug is a hardware error. Given that the OS if
> 5 years old, I'm guessing the hardware is at least that old. I'd
> expect 5-year old hardware to be fairly trustworthy, but it may not
> have been properly-tested before going into production.
> 
> If it's all on a single piece of hardware (all 4 Tomcats), I'd
> blame the hardware and look for a speedy replacement
> (properly-tested this time). If it was on different machines, I'd
> suspect a JVM bug.
> 
> -chris
> 
> ---------------------------------------------------------------------
>
> 
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> ::DISCLAIMER:: 
> ______________________________________________________________________
__________________________________________
>
> 
Confidentiality Notice from Dixons Carphone plc (registered in England &
Wales No.07105905) of 1 Portal Way, London, W3 6RS ("Dixons Carphone").
The information contained in this e-mail and any attachments may be
legally privileged, proprietary and/or confidential. If you received
this e-mail in error, please notify the sender by return, permanently
delete the e-mail and destroy all hard copies immediately. No warranty
is made as to the completeness or accuracy of the information contained
in this e-mail. Opinions, conclusions and statements of intent in this
e-mail are those of the sender and will not bind any Dixons Carphone
group company (Dixons Carphone Group) unless confirmed by an authorised
representative independently of this e-mail. We do not accept
responsibility for viruses; you must scan for these. E-mails sent to and
from Dixons Carphone Group are routinely monitored for record keeping,
quality control, training purposes, to ensure regulatory compliance and
to prevent viruses and unauthorised use of our computer systems. The
Carphone Warehouse Limited (registered in England & Wales No.02142673)
is a member of the Dixons Carphone Group and is authorised and regulated
by the Financial Conduct Authority.
> ______________________________________________________________________
__________________________________________
>
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYimbOAAoJEBzwKT+lPKRYm3EQAL8Vgr7j8o1IPWtFX9U2v3Ce
lXgf6b3mLP+DheBMKKidhSArQGzTa0GmYxwQ0NL4Cx0UYrrlXOsVOJWtzpRZpDKh
Cj0MR9L3pc/2tSiKDCr4f+gxnIfRKrHCJPjU993EQO3WbsqebD85oWEESjJBskVB
tTzbVQTHSvJYbO7rWigH1gTUPM18AqdgqwW2msnvTeUMZ142OIuvt7FzSLI9sW8E
hdQLRyi1u9X4kZUO6Ob+q6cbEt81TLE7goJCNZ7B+PJoxRFN7H4TPlPaYIX9sw20
oOCvySK8dH1BQJ+ufOX+lr8pDpu7vg/PmPCfFcTP705HJioc5G4yI61TIALMq2Qe
a4FImNYbwm07SKQbTSj16PZZ7mKQZqs4UcqdrwJ5qazZXqhTpwgF/qfeu8c1d8d3
BuwTkLJa6BM5n5VU37rCucOE4twM/9nFTnAgdB4kbfHxa6pZzXF9sFvTmPvCLBci
CS/sOcEGbsg1kkHi1iDVFOnhafbG9hGg/alJsnWNJc7XdGXwiL/0Dk64/5HKqAs6
LT/4AYhG/+bwsuEbIKP2zOt2d9fl7NkaI8LJLLgVoAx/JFcLa3+5K4SoM3xycM/Y
aq6TEsMM/IsKoApTkTBWzUEK0lCNPC7MzzBdQQ3bZqmm5VYb2zwCU9TgPi6GnxMx
66GWAL1NlOVQuffsl8D/
=Wkrw
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Apache Tomcat/7.0.39 crashed with fatal error

Posted by Satish Chhatpar 02 <Ch...@cpwplc.com>.
Thanks Chris. I appreciate your help.


All 4 tomcats are on diff machines. One on each, with same tomcat version, same java version and same OS for all.


Tomcats are not in cluster.


Regards

Satish Chhatpar


________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Friday, January 27, 2017 1:52:47 AM
To: Tomcat Users List
Subject: Re: Apache Tomcat/7.0.39 crashed with fatal error

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Satish,

On 1/26/17 2:28 PM, Satish Chhatpar 02 wrote:
> we are using Apache Tomcat/7.0.39 for our java application.

I highly recommend an upgrade for both Tomcat and Java. There are
published vulnerabilities for both product versions you are using.

> There are 4 tomcat instances using same tomcat version and java
> version. yesterday all 4 tomcats crashed with below error in
> hs_err_pid log file.

All on the same hardware? Or separate machines?

> This log file was created for all 4 tomcats.
>
> Its very peculiar behaviour that all 4 crashed around same time.

If they are in a cluster, one going down could cause the load on the
others to go up, increasing the chances of a problem.

> Any information can help us to mitigate this incident.
>
> Apache Tomcat/7.0.39

Unless this is a package-managed version of Tomcat with an
unfortunately inaccurate version number, that version of Tomcat is
nearly 3 years old. The current version in the 7.0.x line is 7.0.75
(released yesterday).

> java version "1.7.0_21" Java(TM) SE Runtime Environment (build
> 1.7.0_21-b11) Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01,
> mixed mode)

That version of Java is also nearly 3 years old. Latest 1.7 build is
1.7.0_80 release nearly 3 years ago. Note that Java 7 is no longer
supported unless you have a long-term support contract with Oracle, in
which case the latest version is 1.7.0_131, released earlier this month.

> OS used
>
>
> Red Hat Enterprise Linux Server release 6.3 (Santiago)

Ouch! 5 years old!

> # # A fatal error has been detected by the Java Runtime
> Environment: # #  SIGSEGV (0xb) at pc=0x00007fed24ecfe9e,
> pid=21352, tid=140656275650304 # # JRE version: 7.0_21-b11 # Java
> VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode
> linux-amd64 compressed oops) # Problematic frame: # J
> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/Lis
t;)Ljava/util/List;
>
>
#
> # Failed to write core dump. Core dumps have been disabled. To
> enable core dumping, try "ulimit -c unlimited" before starting Java
> again # # If you would like to submit a bug report, please visit: #
> http://bugreport.sun.com/bugreport/crash.jsp #

This is either a JVM bug is a hardware error. Given that the OS if 5
years old, I'm guessing the hardware is at least that old. I'd expect
5-year old hardware to be fairly trustworthy, but it may not have been
properly-tested before going into production.

If it's all on a single piece of hardware (all 4 Tomcats), I'd blame
the hardware and look for a speedy replacement (properly-tested this
time). If it was on different machines, I'd suspect a JVM bug.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYilqXAAoJEBzwKT+lPKRYk8gQAJxyGn+KWUb6V0fzWdiF0gzi
xh5p9tjmszS7ixEkMNZmkJH5cwWL8SjrXy0bO3s/50bfzEAOIKnqii+AEBWvKSsw
W/zmh/OFJzONOPhSWFvzkI1q9pND+gwRqyIVIkHYMs/l0Vi8GeqoWwJHjaIfSRtG
ZW9mDg1cH6p8/QrEynTU/B9XzNxKwIxg7YdUu97/uYPXVICDoRyF6Z8y+ISPMSpF
6OT36NDpdQXTL3xWnYM9149HE37dXjAiz1T+ms0FvP4UKLWN8hy58OdUnjW9EQpj
5IpsruGyRKsAKZzUfwWS68t4d2pq8/QUSvi+t23ZVnOkvdUy99thJ+rIUigb2942
NN6mMzpexOxkq36S7DyJrfXUVWdseeoYgEBFz+9mso2TaTf+9RD4JgLIPXjJ2LZp
CvUuHZ1G45uP62qgEsek4LQdPQ8urOzPuDvW578JleRXxE0w80eNPlnFfjb5mFmz
TrjGMWZE9ErnfFu6WB0ef5aNpGC5vh3TMXvT5NCVZvqRz/cBP4cXape0CZZdxxkf
/lfXqOWb0fBGrzS6BoeI44nyL+Nyqei9UZDl8RZaWjah1aGE9bGYSuvvhWXhgorb
Fs/4yvllgIpjRx9gDVcreE0oUFtPYh+FWLSEBYETjEFZCIZVsAGUNNDKa4kDDtXK
pxsK5m9udi1kAN+sqBzC
=lACD
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

::DISCLAIMER::
________________________________________________________________________________________________________________
Confidentiality Notice from Dixons Carphone plc (registered in England & Wales No.07105905) of 1 Portal Way, London, W3 6RS ("Dixons Carphone"). The information contained in this e-mail and any attachments may be legally privileged, proprietary and/or confidential. If you received this e-mail in error, please notify the sender by return, permanently delete the e-mail and destroy all hard copies immediately. No warranty is made as to the completeness or accuracy of the information contained in this e-mail. Opinions, conclusions and statements of intent in this e-mail are those of the sender and will not bind any Dixons Carphone group company (Dixons Carphone Group) unless confirmed by an authorised representative independently of this e-mail. We do not accept responsibility for viruses; you must scan for these. E-mails sent to and from Dixons Carphone Group are routinely monitored for record keeping, quality control, training purposes, to ensure regulatory compliance and to prevent viruses and unauthorised use of our computer systems. The Carphone Warehouse Limited (registered in England & Wales No.02142673) is a member of the Dixons Carphone Group and is authorised and regulated by the Financial Conduct Authority.
________________________________________________________________________________________________________________

Re: Apache Tomcat/7.0.39 crashed with fatal error

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Satish,

On 1/26/17 2:28 PM, Satish Chhatpar 02 wrote:
> we are using Apache Tomcat/7.0.39 for our java application.

I highly recommend an upgrade for both Tomcat and Java. There are
published vulnerabilities for both product versions you are using.

> There are 4 tomcat instances using same tomcat version and java 
> version. yesterday all 4 tomcats crashed with below error in 
> hs_err_pid log file.

All on the same hardware? Or separate machines?

> This log file was created for all 4 tomcats.
> 
> Its very peculiar behaviour that all 4 crashed around same time.

If they are in a cluster, one going down could cause the load on the
others to go up, increasing the chances of a problem.

> Any information can help us to mitigate this incident.
> 
> Apache Tomcat/7.0.39

Unless this is a package-managed version of Tomcat with an
unfortunately inaccurate version number, that version of Tomcat is
nearly 3 years old. The current version in the 7.0.x line is 7.0.75
(released yesterday).

> java version "1.7.0_21" Java(TM) SE Runtime Environment (build
> 1.7.0_21-b11) Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01,
> mixed mode)

That version of Java is also nearly 3 years old. Latest 1.7 build is
1.7.0_80 release nearly 3 years ago. Note that Java 7 is no longer
supported unless you have a long-term support contract with Oracle, in
which case the latest version is 1.7.0_131, released earlier this month.

> OS used
> 
> 
> Red Hat Enterprise Linux Server release 6.3 (Santiago)

Ouch! 5 years old!

> # # A fatal error has been detected by the Java Runtime
> Environment: # #  SIGSEGV (0xb) at pc=0x00007fed24ecfe9e,
> pid=21352, tid=140656275650304 # # JRE version: 7.0_21-b11 # Java
> VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode
> linux-amd64 compressed oops) # Problematic frame: # J
> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/Lis
t;)Ljava/util/List;
>
> 
#
> # Failed to write core dump. Core dumps have been disabled. To
> enable core dumping, try "ulimit -c unlimited" before starting Java
> again # # If you would like to submit a bug report, please visit: #
> http://bugreport.sun.com/bugreport/crash.jsp #

This is either a JVM bug is a hardware error. Given that the OS if 5
years old, I'm guessing the hardware is at least that old. I'd expect
5-year old hardware to be fairly trustworthy, but it may not have been
properly-tested before going into production.

If it's all on a single piece of hardware (all 4 Tomcats), I'd blame
the hardware and look for a speedy replacement (properly-tested this
time). If it was on different machines, I'd suspect a JVM bug.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYilqXAAoJEBzwKT+lPKRYk8gQAJxyGn+KWUb6V0fzWdiF0gzi
xh5p9tjmszS7ixEkMNZmkJH5cwWL8SjrXy0bO3s/50bfzEAOIKnqii+AEBWvKSsw
W/zmh/OFJzONOPhSWFvzkI1q9pND+gwRqyIVIkHYMs/l0Vi8GeqoWwJHjaIfSRtG
ZW9mDg1cH6p8/QrEynTU/B9XzNxKwIxg7YdUu97/uYPXVICDoRyF6Z8y+ISPMSpF
6OT36NDpdQXTL3xWnYM9149HE37dXjAiz1T+ms0FvP4UKLWN8hy58OdUnjW9EQpj
5IpsruGyRKsAKZzUfwWS68t4d2pq8/QUSvi+t23ZVnOkvdUy99thJ+rIUigb2942
NN6mMzpexOxkq36S7DyJrfXUVWdseeoYgEBFz+9mso2TaTf+9RD4JgLIPXjJ2LZp
CvUuHZ1G45uP62qgEsek4LQdPQ8urOzPuDvW578JleRXxE0w80eNPlnFfjb5mFmz
TrjGMWZE9ErnfFu6WB0ef5aNpGC5vh3TMXvT5NCVZvqRz/cBP4cXape0CZZdxxkf
/lfXqOWb0fBGrzS6BoeI44nyL+Nyqei9UZDl8RZaWjah1aGE9bGYSuvvhWXhgorb
Fs/4yvllgIpjRx9gDVcreE0oUFtPYh+FWLSEBYETjEFZCIZVsAGUNNDKa4kDDtXK
pxsK5m9udi1kAN+sqBzC
=lACD
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org