You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Bjørn T Johansen <bt...@havleik.no> on 2014/11/08 23:08:58 UTC

Problem when rebooting....

I have a init script I have been using now for severeal years that has been working without any problems...
But since upgrading to Tomcat 8 (now 8.0.14), Tomcat is not coming up if I boot the server..

Here is the output from catalina.out..:

08-Nov-2014 23:03:09.657 INFO [Thread-4] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-apr-8080"]
08-Nov-2014 23:03:09.724 INFO [Thread-4] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-apr-8009"]
08-Nov-2014 23:03:09.775 INFO [Thread-4] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
08-Nov-2014 23:03:10.089 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [/kappskole] appears to have started a thread named [Mojarra-WebResourceMonitor-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)
08-Nov-2014 23:03:10.097 INFO [localhost-startStop-2] org.springframework.context.support.AbstractApplicationContext.doClose Closing Root WebApplicationContext: startup date [Sat Nov 01 23:59:29 CET 2014]; root of context hierarchy
08-Nov-2014 23:03:10.104 INFO [localhost-startStop-2] org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7b584330: defining beans [saldo,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,dataSource,jdbcTemplate]; root of factory hierarchy
08-Nov-2014 23:03:10.128 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [/trumf] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
08-Nov-2014 23:03:10.164 INFO [Thread-4] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-apr-8080"]
08-Nov-2014 23:03:10.216 INFO [Thread-4] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-apr-8009"]
08-Nov-2014 23:03:10.294 INFO [Thread-4] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-apr-8080"]
08-Nov-2014 23:03:10.295 INFO [Thread-4] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-apr-8009"]
08-Nov-2014 23:03:41.930 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.0.14
08-Nov-2014 23:03:41.941 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:   Sep 24 2014 09:01:51
08-Nov-2014 23:03:41.941 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:  8.0.14.0
08-Nov-2014 23:03:41.942 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:        Linux
08-Nov-2014 23:03:41.942 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:     2.6.32-504.el6.x86_64
08-Nov-2014 23:03:41.942 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:   amd64
08-Nov-2014 23:03:41.943 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:    1.8.0_25-b17
08-Nov-2014 23:03:41.943 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:     Oracle Corporation
08-Nov-2014 23:03:43.325 INFO [main] org.apache.catalina.core.AprLifecycleListener.init Loaded APR based Apache Tomcat Native library 1.1.31 using APR version 1.3.9.
08-Nov-2014 23:03:43.326 INFO [main] org.apache.catalina.core.AprLifecycleListener.init APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
08-Nov-2014 23:03:43.651 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized (OpenSSL 1.0.1e 11 Feb 2013)
08-Nov-2014 23:03:44.102 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-apr-8080"]
08-Nov-2014 23:03:44.135 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-apr-8009"]
08-Nov-2014 23:03:44.169 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 2655 ms
08-Nov-2014 23:03:44.561 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
08-Nov-2014 23:03:44.561 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.14
08-Nov-2014 23:03:44.666 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application
archive /lvm/vg1-data/opt/apache-tomcat-8.0.14/webapps/vegaappen.war

As you can see, tomcat just stops at the first web application....

What am I missing?



Regards,

BTJ

-- 
-----------------------------------------------------------------------------------------------
Bjørn T Johansen

btj@havleik.no
-----------------------------------------------------------------------------------------------
Someone wrote:
"I understand that if you play a Windows CD backwards you hear strange Satanic messages"
To which someone replied:
"It's even worse than that; play it forwards and it installs Windows"
-----------------------------------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Problem when rebooting....

Posted by Bjørn T Johansen <bt...@havleik.no>.
On Mon, 10 Nov 2014 09:08:56 -0500
Christopher Schultz <ch...@christopherschultz.net> wrote:

> 
> Perhaps something is being read from a network filesystem that has
> stalled mounting or something similar? The question for the OP is
> "what is different during boot time?"
> 
> - -chris

Nope, no such things... All is based on local filesystem... And tried to think what might be different and the only thing that is usually different,
is that I delete anything beneath the work directory... So I added deletion to the startup script but Tomcat still hangs....
I am lost to what else may be different in starting tomcat from the init script or starting tomcat from the init script from the command line....

BTJ

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Problem when rebooting....

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ameer,

On 11/9/14 11:42 PM, Ameer Mawia wrote:
> I'm looking into the issue(I had been longing to contribute to
> community and I see some opportunity to start doing so here. Hope
> my effort will be welcomed!:) )

Absolutely. Thanks for your contributions.

> I'm looking at the code base in trunk. Here is the method where 
> localhost-startstop thread is busy:
> 
> 
> protected void processAnnotationsFile(File file, WebXml fragment,
> //line 1963 boolean handlesTypesOnly) {
> 
> if (file.isDirectory()) { // Returns null if directory is not
> readable String[] dirs = file.list();//line 1968 if (dirs != null)
> { for (String dir : dirs) { processAnnotationsFile(    //line 1971 
> new File(file,dir), fragment, handlesTypesOnly); } } } else if
> (file.canRead() && file.getName().endsWith(".class")) { try
> (FileInputStream fis = new FileInputStream(file)) { 
> processAnnotationsStream(fis, fragment, handlesTypesOnly); } catch
> (IOException e) { 
> log.error(sm.getString("contextConfig.inputStreamFile", 
> file.getAbsolutePath()),e); } catch (ClassFormatException e) { 
> log.error(sm.getString("contextConfig.inputStreamFile", 
> file.getAbsolutePath()),e); } } }
> 
> As you can see and as konstantin said, the thread is busy walking
> over directories listed in web-fragements.

Here, the canRead call is being done in such a way that entropy should
not be a part of the problem.

Perhaps something is being read from a network filesystem that has
stalled mounting or something similar? The question for the OP is
"what is different during boot time?"

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJUYMb4AAoJEBzwKT+lPKRYjwUP/jLSdAXlD67RR+UYe0NhPbF8
GweAbj/zDtuLq3enElNa838LnlVoG/yKKvIkA8cQ2GgdtjP0PKzJeIUqSRBVuQ6m
gYg/iiY2qgi6tKfOsEUobGhZrUa8xfzVigASyfKwxVnXE3kShKprY02iq0U7eURW
33cGOgezcq4CvEm4MRcLfBGJqlYtjRWLvWQGt+u3u6IqJ+1gZ2v9VFgembDu6Fjo
2AdRWNGiJGvIPPKeEU4PShk5uWliLdTCToRjCJ6Xm0q9GKIPY6oE616b+sLVz9AP
q1Fzq7ZgMKkwIIcCGsbBLurKm5lmizB5cTCQm870kh2C0fY4I5HXthKwADvBBcn8
eU/0UPe6z2Mn8pgzTBYXTDCH8F8JdRAqVkL//WU0vaZEqNVKxYkbPJZjseYerEjS
huv8N1kXl8GNueWBJ5H+N0VUfrLJOwzOeSRYx8svxF1p79xGOxF2rmKhypLLIPpL
ldoLWFnJ2P1dDPTDEkdsxYJQxZ0FB9eDcDgIH2gGDUuOcyUW7xqK4kCfzo9Yr4MX
c5iIwtsGz0/mELSdY30/Qx2lflEL0NvDj54NIz6o8pDVsIGvSWmV5RgaM9Bb7+84
oehx5TC4GzHTEgrBvU292NH2UQf1lXCPCMYHI15tbEOIBX2NjbqKJp+o/1+ajOl4
IVIII1t4+kGs7yAhL5Pa
=D5Gs
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Problem when rebooting....

Posted by Ameer Mawia <am...@gmail.com>.
Hi Bjorn,

I'm looking into the issue(I had been longing to contribute to community
and I see some opportunity to start doing so here. Hope my effort will be
welcomed!:) )


I'm looking at the code base in trunk. Here is the method where
localhost-startstop thread is busy:


   protected void processAnnotationsFile(File file, WebXml fragment, //line
1963
            boolean handlesTypesOnly) {

        if (file.isDirectory()) {
            // Returns null if directory is not readable
            String[] dirs = file.list();//line 1968
            if (dirs != null) {
                for (String dir : dirs) {
                    processAnnotationsFile(    //line 1971
                            new File(file,dir), fragment, handlesTypesOnly);
                }
            }
        } else if (file.canRead() && file.getName().endsWith(".class")) {
            try (FileInputStream fis = new FileInputStream(file)) {
                processAnnotationsStream(fis, fragment, handlesTypesOnly);
            } catch (IOException e) {
                log.error(sm.getString("contextConfig.inputStreamFile",
                        file.getAbsolutePath()),e);
            } catch (ClassFormatException e) {
                log.error(sm.getString("contextConfig.inputStreamFile",
                        file.getAbsolutePath()),e);
            }
        }
    }

As you can see and as konstantin said, the thread is busy walking over
directories listed in web-fragements. Here it is from where it is being
called:

 protected void processAnnotations(Set<WebXml> fragments,
            boolean handlesTypesOnly) {
        for(WebXml fragment : fragments) {
            // Only need to scan for @HandlesTypes matches if any of the
            // following are true:
            // - it has already been determined only @HandlesTypes is
required
            //   (e.g. main web.xml has metadata-complete="true"
            // - this fragment is for a container JAR (Servlet 3.1 section
8.1)
            // - this fragment has metadata-complete="true"
            boolean htOnly = handlesTypesOnly || !fragment.getWebappJar() ||
                    fragment.isMetadataComplete();

            WebXml annotations = new WebXml();
            // no impact on distributable
            annotations.setDistributable(true);
            URL url = fragment.getURL();
            processAnnotationsUrl(url, annotations, htOnly);// This in turn
calls processAnnotationsFile
            Set<WebXml> set = new HashSet<>();
            set.add(annotations);
            // Merge annotations into fragment - fragment takes priority
            fragment.merge(set);
        }
    }

I'will let you know, as soon as I come to some concrete understanding.
Meanwhile you can take a look at various web-fragments in your
web-application.

Regards,
Ameer Mawia,
New Delhi

On Mon, Nov 10, 2014 at 4:55 AM, Bjørn T Johansen <bt...@havleik.no> wrote:

> On Sun, 09 Nov 2014 23:49:54 +0100
> André Warnier <aw...@ice-sa.com> wrote:
>
> > Bjørn T Johansen wrote:
> > > On Mon, 10 Nov 2014 00:35:46 +0300
> > > Konstantin Kolinko <kn...@gmail.com> wrote:
> > >
> > >> Rules:
> > >> http://tomcat.apache.org/lists.html#tomcat-users
> > >> -> 6. Do not top-post.
> > >>
> > >>> Ok, did a thread dump that resulted in the following dump..:
> > >>>
> > >>>
> > >>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed
> mode):
> > >>>
> > >>> "localhost-startStop-1" #15 daemon prio=5 os_prio=0
> tid=0x00007fdc84002000 nid=0x702 runnable [0x00007fdc8ebe7000]
> > >>>    java.lang.Thread.State: RUNNABLE
> > >>>         at java.io.UnixFileSystem.checkAccess(Native Method)
> > >>>         at java.io.File.canRead(File.java:768)
> > >>>         at
> org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1975)
> > >>>         at
> org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
> > >>
> > >> As it is visible from the method name, the "start" thread is busy
> > >> scanning classes for annotations.
> > >>
> > >> This behaviour is explained here:
> > >>
> > >>>> https://wiki.apache.org/tomcat/HowTo/FasterStartUp#General
> > >
> > > But it never finishes? And also I do not have this problem when
> stopping and starting the Tomcat server without booting, why is that?
> Shouldn't the
> > > result be the same?
> > >
> >
> > http://en.wikipedia.org/wiki/Entropy_%28computing%29
> > Maybe your problem is there ?
> > (When the system boots, it has little entropy available, and it may
> already be consumed by
> > other processes starting around the same time).
> >
>
> Hmm... Guess I'll havce to check it out... Thx... :)
>
> BTJ
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: Problem when rebooting....

Posted by Bjørn T Johansen <bt...@havleik.no>.
On Sun, 09 Nov 2014 23:49:54 +0100
André Warnier <aw...@ice-sa.com> wrote:

> Bjørn T Johansen wrote:
> > On Mon, 10 Nov 2014 00:35:46 +0300
> > Konstantin Kolinko <kn...@gmail.com> wrote:
> > 
> >> Rules:
> >> http://tomcat.apache.org/lists.html#tomcat-users
> >> -> 6. Do not top-post.
> >>
> >>> Ok, did a thread dump that resulted in the following dump..:
> >>>
> >>>
> >>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode):
> >>>
> >>> "localhost-startStop-1" #15 daemon prio=5 os_prio=0 tid=0x00007fdc84002000 nid=0x702 runnable [0x00007fdc8ebe7000]
> >>>    java.lang.Thread.State: RUNNABLE
> >>>         at java.io.UnixFileSystem.checkAccess(Native Method)
> >>>         at java.io.File.canRead(File.java:768)
> >>>         at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1975)
> >>>         at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
> >>
> >> As it is visible from the method name, the "start" thread is busy
> >> scanning classes for annotations.
> >>
> >> This behaviour is explained here:
> >>
> >>>> https://wiki.apache.org/tomcat/HowTo/FasterStartUp#General
> > 
> > But it never finishes? And also I do not have this problem when stopping and starting the Tomcat server without booting, why is that? Shouldn't the
> > result be the same?
> > 
> 
> http://en.wikipedia.org/wiki/Entropy_%28computing%29
> Maybe your problem is there ?
> (When the system boots, it has little entropy available, and it may already be consumed by 
> other processes starting around the same time).
> 

Hmm... Guess I'll havce to check it out... Thx... :)

BTJ

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Problem when rebooting....

Posted by André Warnier <aw...@ice-sa.com>.
Bjørn T Johansen wrote:
> On Mon, 10 Nov 2014 00:35:46 +0300
> Konstantin Kolinko <kn...@gmail.com> wrote:
> 
>> Rules:
>> http://tomcat.apache.org/lists.html#tomcat-users
>> -> 6. Do not top-post.
>>
>>> Ok, did a thread dump that resulted in the following dump..:
>>>
>>>
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode):
>>>
>>> "localhost-startStop-1" #15 daemon prio=5 os_prio=0 tid=0x00007fdc84002000 nid=0x702 runnable [0x00007fdc8ebe7000]
>>>    java.lang.Thread.State: RUNNABLE
>>>         at java.io.UnixFileSystem.checkAccess(Native Method)
>>>         at java.io.File.canRead(File.java:768)
>>>         at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1975)
>>>         at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
>>
>> As it is visible from the method name, the "start" thread is busy
>> scanning classes for annotations.
>>
>> This behaviour is explained here:
>>
>>>> https://wiki.apache.org/tomcat/HowTo/FasterStartUp#General
> 
> But it never finishes? And also I do not have this problem when stopping and starting the Tomcat server without booting, why is that? Shouldn't the
> result be the same?
> 

http://en.wikipedia.org/wiki/Entropy_%28computing%29
Maybe your problem is there ?
(When the system boots, it has little entropy available, and it may already be consumed by 
other processes starting around the same time).


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Problem when rebooting....

Posted by Bjørn T Johansen <bt...@havleik.no>.
On Mon, 10 Nov 2014 00:35:46 +0300
Konstantin Kolinko <kn...@gmail.com> wrote:

> Rules:
> http://tomcat.apache.org/lists.html#tomcat-users
> -> 6. Do not top-post.
> 
> > Ok, did a thread dump that resulted in the following dump..:
> >
> >
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode):
> >
> > "localhost-startStop-1" #15 daemon prio=5 os_prio=0 tid=0x00007fdc84002000 nid=0x702 runnable [0x00007fdc8ebe7000]
> >    java.lang.Thread.State: RUNNABLE
> >         at java.io.UnixFileSystem.checkAccess(Native Method)
> >         at java.io.File.canRead(File.java:768)
> >         at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1975)
> >         at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
> 
> 
> As it is visible from the method name, the "start" thread is busy
> scanning classes for annotations.
> 
> This behaviour is explained here:
> 
> >> https://wiki.apache.org/tomcat/HowTo/FasterStartUp#General
> 

But it never finishes? And also I do not have this problem when stopping and starting the Tomcat server without booting, why is that? Shouldn't the
result be the same?


BTJ

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Problem when rebooting....

Posted by Konstantin Kolinko <kn...@gmail.com>.
Rules:
http://tomcat.apache.org/lists.html#tomcat-users
-> 6. Do not top-post.

> Ok, did a thread dump that resulted in the following dump..:
>
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode):
>
> "localhost-startStop-1" #15 daemon prio=5 os_prio=0 tid=0x00007fdc84002000 nid=0x702 runnable [0x00007fdc8ebe7000]
>    java.lang.Thread.State: RUNNABLE
>         at java.io.UnixFileSystem.checkAccess(Native Method)
>         at java.io.File.canRead(File.java:768)
>         at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1975)
>         at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)


As it is visible from the method name, the "start" thread is busy
scanning classes for annotations.

This behaviour is explained here:

>> https://wiki.apache.org/tomcat/HowTo/FasterStartUp#General

Best regards,
Konstantin Kolinko

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Problem when rebooting....

Posted by Bjørn T Johansen <bt...@havleik.no>.
Ok, did a thread dump that resulted in the following dump..:


Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode):

"localhost-startStop-1" #15 daemon prio=5 os_prio=0 tid=0x00007fdc84002000 nid=0x702 runnable [0x00007fdc8ebe7000]
   java.lang.Thread.State: RUNNABLE
	at java.io.UnixFileSystem.checkAccess(Native Method)
	at java.io.File.canRead(File.java:768)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1975)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1922)
	at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1880)
	at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1149)
	at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:771)
	- locked <0x00000000c1395170> (a org.apache.catalina.startup.ContextConfig)
	at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:305)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5120)
	- locked <0x00000000c1394f40> (a org.apache.catalina.core.StandardContext)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	- locked <0x00000000c1394f40> (a org.apache.catalina.core.StandardContext)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:724)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:714)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:919)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1703)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"Catalina-startStop-1" #14 daemon prio=5 os_prio=0 tid=0x00007fdce4499000 nid=0x6fe waiting on condition [0x00007fdc8ece9000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c1292150> (a java.util.concurrent.FutureTask)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:720)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:396)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1438)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:282)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:402)
	- locked <0x00000000c12921d0> (a org.apache.catalina.core.StandardHost)
	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:347)
	- locked <0x00000000c12921d0> (a org.apache.catalina.core.StandardHost)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:933)
	- locked <0x00000000c12921d0> (a org.apache.catalina.core.StandardHost)
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:868)
	- locked <0x00000000c12921d0> (a org.apache.catalina.core.StandardHost)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	- locked <0x00000000c12921d0> (a org.apache.catalina.core.StandardHost)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"Tomcat JDBC Pool Cleaner[321001045:1415490437332]" #13 daemon prio=5 os_prio=0 tid=0x00007fdce4452000 nid=0x6fd in Object.wait() [0x00007fdc8efeb000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	- locked <0x00000000c07c4a10> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

"GC Daemon" #12 daemon prio=2 os_prio=0 tid=0x00007fdce438f800 nid=0x6f4 in Object.wait() [0x00007fdc8f549000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000c02a43a0> (a sun.misc.GC$LatencyLock)
	at sun.misc.GC$Daemon.run(GC.java:117)
	- locked <0x00000000c02a43a0> (a sun.misc.GC$LatencyLock)

"AsyncFileHandlerWriter-225534817" #11 daemon prio=5 os_prio=0 tid=0x00007fdce4104800 nid=0x67e waiting on condition [0x00007fdcd0178000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c1352e98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.juli.AsyncFileHandler$LoggerThread.run(AsyncFileHandler.java:144)

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fdce40b0800 nid=0x65b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fdce40ad800 nid=0x658 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fdce40ab800 nid=0x656 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fdce40a8800 nid=0x655 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fdce40a6800 nid=0x654 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fdce407b800 nid=0x640 in Object.wait() [0x00007fdcd087f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
	- locked <0x00000000c1290638> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fdce4079800 nid=0x63f in Object.wait() [0x00007fdcd0980000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
	- locked <0x00000000c02a54e0> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x00007fdce4009000 nid=0x634 waiting on condition [0x00007fdceaa28000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c07c57e8> (a java.util.concurrent.FutureTask)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
	- locked <0x00000000c139b980> (a org.apache.catalina.core.StandardEngine)
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
	- locked <0x00000000c139b980> (a org.apache.catalina.core.StandardEngine)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	- locked <0x00000000c139b980> (a org.apache.catalina.core.StandardEngine)
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:439)
	- locked <0x00000000c139b980> (a org.apache.catalina.core.StandardEngine)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	- locked <0x00000000c1399e20> (a org.apache.catalina.core.StandardService)
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:760)
	- locked <0x00000000c0815fd8> (a java.lang.Object)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	- locked <0x00000000c1354430> (a org.apache.catalina.core.StandardServer)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:351)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:485)

"VM Thread" os_prio=0 tid=0x00007fdce4072000 nid=0x63d runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fdce401e000 nid=0x638 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fdce4020000 nid=0x639 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fdce4021800 nid=0x63a runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fdce4023800 nid=0x63b runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007fdce40b7800 nid=0x65e waiting on condition 

JNI global references: 63

Heap
 PSYoungGen      total 295424K, used 237522K [0x00000000eab00000, 0x00000000ff200000, 0x0000000100000000)
  eden space 283648K, 83% used [0x00000000eab00000,0x00000000f92d4ba0,0x00000000fc000000)
  from space 11776K, 1% used [0x00000000fe680000,0x00000000fe6a0000,0x00000000ff200000)
  to   space 25600K, 0% used [0x00000000fc000000,0x00000000fc000000,0x00000000fd900000)
 ParOldGen       total 62976K, used 44939K [0x00000000c0000000, 0x00000000c3d80000, 0x00000000eab00000)
  object space 62976K, 71% used [0x00000000c0000000,0x00000000c2be2ff8,0x00000000c3d80000)
 Metaspace       used 17286K, capacity 17546K, committed 17792K, reserved 1064960K
  class space    used 2072K, capacity 2142K, committed 2176K, reserved 1048576K

2014-11-09 00:48:43
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode):

"localhost-startStop-1" #15 daemon prio=5 os_prio=0 tid=0x00007fdc84002000 nid=0x702 runnable [0x00007fdc8ebe7000]
   java.lang.Thread.State: RUNNABLE
	at java.io.UnixFileSystem.list(Native Method)
	at java.io.File.list(File.java:1122)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1968)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsFile(ContextConfig.java:1971)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1922)
	at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1880)
	at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1149)
	at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:771)
	- locked <0x00000000c1395170> (a org.apache.catalina.startup.ContextConfig)
	at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:305)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5120)
	- locked <0x00000000c1394f40> (a org.apache.catalina.core.StandardContext)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	- locked <0x00000000c1394f40> (a org.apache.catalina.core.StandardContext)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:724)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:714)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:919)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1703)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"Catalina-startStop-1" #14 daemon prio=5 os_prio=0 tid=0x00007fdce4499000 nid=0x6fe waiting on condition [0x00007fdc8ece9000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c1292150> (a java.util.concurrent.FutureTask)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:720)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:396)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1438)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:282)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:402)
	- locked <0x00000000c12921d0> (a org.apache.catalina.core.StandardHost)
	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:347)
	- locked <0x00000000c12921d0> (a org.apache.catalina.core.StandardHost)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:933)
	- locked <0x00000000c12921d0> (a org.apache.catalina.core.StandardHost)
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:868)
	- locked <0x00000000c12921d0> (a org.apache.catalina.core.StandardHost)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	- locked <0x00000000c12921d0> (a org.apache.catalina.core.StandardHost)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"Tomcat JDBC Pool Cleaner[321001045:1415490437332]" #13 daemon prio=5 os_prio=0 tid=0x00007fdce4452000 nid=0x6fd in Object.wait() [0x00007fdc8efeb000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	- locked <0x00000000c07c4a10> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

"GC Daemon" #12 daemon prio=2 os_prio=0 tid=0x00007fdce438f800 nid=0x6f4 in Object.wait() [0x00007fdc8f549000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000c02a43a0> (a sun.misc.GC$LatencyLock)
	at sun.misc.GC$Daemon.run(GC.java:117)
	- locked <0x00000000c02a43a0> (a sun.misc.GC$LatencyLock)

"AsyncFileHandlerWriter-225534817" #11 daemon prio=5 os_prio=0 tid=0x00007fdce4104800 nid=0x67e waiting on condition [0x00007fdcd0178000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c1352e98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.juli.AsyncFileHandler$LoggerThread.run(AsyncFileHandler.java:144)

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fdce40b0800 nid=0x65b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fdce40ad800 nid=0x658 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fdce40ab800 nid=0x656 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fdce40a8800 nid=0x655 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fdce40a6800 nid=0x654 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fdce407b800 nid=0x640 in Object.wait() [0x00007fdcd087f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
	- locked <0x00000000c1290638> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fdce4079800 nid=0x63f in Object.wait() [0x00007fdcd0980000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
	- locked <0x00000000c02a54e0> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x00007fdce4009000 nid=0x634 waiting on condition [0x00007fdceaa28000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c07c57e8> (a java.util.concurrent.FutureTask)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
	- locked <0x00000000c139b980> (a org.apache.catalina.core.StandardEngine)
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
	- locked <0x00000000c139b980> (a org.apache.catalina.core.StandardEngine)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	- locked <0x00000000c139b980> (a org.apache.catalina.core.StandardEngine)
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:439)
	- locked <0x00000000c139b980> (a org.apache.catalina.core.StandardEngine)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	- locked <0x00000000c1399e20> (a org.apache.catalina.core.StandardService)
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:760)
	- locked <0x00000000c0815fd8> (a java.lang.Object)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	- locked <0x00000000c1354430> (a org.apache.catalina.core.StandardServer)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:351)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:485)

"VM Thread" os_prio=0 tid=0x00007fdce4072000 nid=0x63d runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fdce401e000 nid=0x638 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fdce4020000 nid=0x639 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fdce4021800 nid=0x63a runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fdce4023800 nid=0x63b runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007fdce40b7800 nid=0x65e waiting on condition 

JNI global references: 63

Heap
 PSYoungGen      total 220160K, used 86535K [0x00000000eab00000, 0x00000000fa700000, 0x0000000100000000)
  eden space 216064K, 39% used [0x00000000eab00000,0x00000000eff61d70,0x00000000f7e00000)
  from space 4096K, 3% used [0x00000000fa300000,0x00000000fa320000,0x00000000fa700000)
  to   space 20992K, 0% used [0x00000000f7e00000,0x00000000f7e00000,0x00000000f9280000)
 ParOldGen       total 62976K, used 45019K [0x00000000c0000000, 0x00000000c3d80000, 0x00000000eab00000)
  object space 62976K, 71% used [0x00000000c0000000,0x00000000c2bf6ff8,0x00000000c3d80000)
 Metaspace       used 17286K, capacity 17546K, committed 17792K, reserved 1064960K
  class space    used 2072K, capacity 2142K, committed 2176K, reserved 1048576K



Not sure how to interpret this, any ideas?

I have tried waiting for a loong time but the tomcat server never comes up to respond to tcp traffic....


BTJ

On Sun, 9 Nov 2014 02:20:15 +0300
Konstantin Kolinko <kn...@gmail.com> wrote:

> 2014-11-09 1:08 GMT+03:00 Bjørn T Johansen <bt...@havleik.no>:
> > I have a init script I have been using now for severeal years that has been working without any problems...
> > But since upgrading to Tomcat 8 (now 8.0.14), Tomcat is not coming up if I boot the server..
> >
> > Here is the output from catalina.out..:
> >
> > 08-Nov-2014 23:03:09.657 INFO [Thread-4] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-apr-8080"]
> > 08-Nov-2014 23:03:09.724 INFO [Thread-4] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-apr-8009"]
> > 08-Nov-2014 23:03:09.775 INFO [Thread-4] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
> > 08-Nov-2014 23:03:10.089 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [/kappskole] appears to have started a thread named [Mojarra-WebResourceMonitor-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
> >  sun.misc.Unsafe.park(Native Method)
> >  java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >  java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >  java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >  java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >  java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >  java.lang.Thread.run(Thread.java:745)
> > 08-Nov-2014 23:03:10.097 INFO [localhost-startStop-2] org.springframework.context.support.AbstractApplicationContext.doClose Closing Root WebApplicationContext: startup date [Sat Nov 01 23:59:29 CET 2014]; root of context hierarchy
> > 08-Nov-2014 23:03:10.104 INFO [localhost-startStop-2] org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7b584330: defining beans [saldo,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,dataSource,jdbcTemplate]; root of factory hierarchy
> > 08-Nov-2014 23:03:10.128 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [/trumf] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
> > 08-Nov-2014 23:03:10.164 INFO [Thread-4] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-apr-8080"]
> > 08-Nov-2014 23:03:10.216 INFO [Thread-4] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-apr-8009"]
> > 08-Nov-2014 23:03:10.294 INFO [Thread-4] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-apr-8080"]
> > 08-Nov-2014 23:03:10.295 INFO [Thread-4] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-apr-8009"]
> > 08-Nov-2014 23:03:41.930 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.0.14
> > 08-Nov-2014 23:03:41.941 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:   Sep 24 2014 09:01:51
> > 08-Nov-2014 23:03:41.941 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:  8.0.14.0
> > 08-Nov-2014 23:03:41.942 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:        Linux
> > 08-Nov-2014 23:03:41.942 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:     2.6.32-504.el6.x86_64
> > 08-Nov-2014 23:03:41.942 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:   amd64
> > 08-Nov-2014 23:03:41.943 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:    1.8.0_25-b17
> > 08-Nov-2014 23:03:41.943 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:     Oracle Corporation
> > 08-Nov-2014 23:03:43.325 INFO [main] org.apache.catalina.core.AprLifecycleListener.init Loaded APR based Apache Tomcat Native library 1.1.31 using APR version 1.3.9.
> > 08-Nov-2014 23:03:43.326 INFO [main] org.apache.catalina.core.AprLifecycleListener.init APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
> > 08-Nov-2014 23:03:43.651 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized (OpenSSL 1.0.1e 11 Feb 2013)
> > 08-Nov-2014 23:03:44.102 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-apr-8080"]
> > 08-Nov-2014 23:03:44.135 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-apr-8009"]
> > 08-Nov-2014 23:03:44.169 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 2655 ms
> > 08-Nov-2014 23:03:44.561 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
> > 08-Nov-2014 23:03:44.561 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.14
> > 08-Nov-2014 23:03:44.666 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application
> > archive /lvm/vg1-data/opt/apache-tomcat-8.0.14/webapps/vegaappen.war
> >
> > As you can see, tomcat just stops at the first web application....
> >
> > What am I missing?
> 
> https://wiki.apache.org/tomcat/HowTo/FasterStartUp#General
> 
> Take thread dumps to see what it is actually doing. It may be
> annotation scanning. It may be waiting for entropy.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Problem when rebooting....

Posted by Konstantin Kolinko <kn...@gmail.com>.
2014-11-09 1:08 GMT+03:00 Bjørn T Johansen <bt...@havleik.no>:
> I have a init script I have been using now for severeal years that has been working without any problems...
> But since upgrading to Tomcat 8 (now 8.0.14), Tomcat is not coming up if I boot the server..
>
> Here is the output from catalina.out..:
>
> 08-Nov-2014 23:03:09.657 INFO [Thread-4] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-apr-8080"]
> 08-Nov-2014 23:03:09.724 INFO [Thread-4] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-apr-8009"]
> 08-Nov-2014 23:03:09.775 INFO [Thread-4] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
> 08-Nov-2014 23:03:10.089 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [/kappskole] appears to have started a thread named [Mojarra-WebResourceMonitor-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
>  sun.misc.Unsafe.park(Native Method)
>  java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>  java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>  java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>  java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>  java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  java.lang.Thread.run(Thread.java:745)
> 08-Nov-2014 23:03:10.097 INFO [localhost-startStop-2] org.springframework.context.support.AbstractApplicationContext.doClose Closing Root WebApplicationContext: startup date [Sat Nov 01 23:59:29 CET 2014]; root of context hierarchy
> 08-Nov-2014 23:03:10.104 INFO [localhost-startStop-2] org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7b584330: defining beans [saldo,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,dataSource,jdbcTemplate]; root of factory hierarchy
> 08-Nov-2014 23:03:10.128 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [/trumf] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
> 08-Nov-2014 23:03:10.164 INFO [Thread-4] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-apr-8080"]
> 08-Nov-2014 23:03:10.216 INFO [Thread-4] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-apr-8009"]
> 08-Nov-2014 23:03:10.294 INFO [Thread-4] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-apr-8080"]
> 08-Nov-2014 23:03:10.295 INFO [Thread-4] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-apr-8009"]
> 08-Nov-2014 23:03:41.930 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.0.14
> 08-Nov-2014 23:03:41.941 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:   Sep 24 2014 09:01:51
> 08-Nov-2014 23:03:41.941 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:  8.0.14.0
> 08-Nov-2014 23:03:41.942 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:        Linux
> 08-Nov-2014 23:03:41.942 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:     2.6.32-504.el6.x86_64
> 08-Nov-2014 23:03:41.942 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:   amd64
> 08-Nov-2014 23:03:41.943 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:    1.8.0_25-b17
> 08-Nov-2014 23:03:41.943 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:     Oracle Corporation
> 08-Nov-2014 23:03:43.325 INFO [main] org.apache.catalina.core.AprLifecycleListener.init Loaded APR based Apache Tomcat Native library 1.1.31 using APR version 1.3.9.
> 08-Nov-2014 23:03:43.326 INFO [main] org.apache.catalina.core.AprLifecycleListener.init APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
> 08-Nov-2014 23:03:43.651 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized (OpenSSL 1.0.1e 11 Feb 2013)
> 08-Nov-2014 23:03:44.102 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-apr-8080"]
> 08-Nov-2014 23:03:44.135 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-apr-8009"]
> 08-Nov-2014 23:03:44.169 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 2655 ms
> 08-Nov-2014 23:03:44.561 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
> 08-Nov-2014 23:03:44.561 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.14
> 08-Nov-2014 23:03:44.666 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application
> archive /lvm/vg1-data/opt/apache-tomcat-8.0.14/webapps/vegaappen.war
>
> As you can see, tomcat just stops at the first web application....
>
> What am I missing?

https://wiki.apache.org/tomcat/HowTo/FasterStartUp#General

Take thread dumps to see what it is actually doing. It may be
annotation scanning. It may be waiting for entropy.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org