You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@tomcat.apache.org by Jerry Malcolm <te...@malcolms.com> on 2019/05/18 19:55:43 UTC

Massive Startup Time after Server Reboot

This is a weird one.  It started a few months ago.   I have TC 9 running 
on Windows Server 16.  After I reboot the entire server, Tomcat takes 
forever on startup.  It normally starts in about 30 seconds.  But after 
a server reboot it takes up to 15 minutes... chugging along at a snail's 
pace starting up all of the apps on all of the virtual hosts.  It always 
finally gets there with everything successfully running.  Other servers 
on the same box (Apache, JAMES, ISC BIND, MySQL) don't have any problem 
starting up.  CPU, Disk, Memory, etc. usages are barely showing on the 
performance graphs.  There's nothing in the Catalina log or system.err 
other than showing a couple of minute gap in the time stamp between each 
app as it starts up.  If I need to reboot again later, it boots again in 
about 30 sec as expected.

First question... any ideas off the top of your head that might be 
causing this?

Second question... is there any other logging I can turn on that might 
explain what TC is blocking on?

Thanks.

Jerry


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Massive Startup Time after Server Reboot

Posted by Suvendu Sekhar Mondal <su...@gmail.com>.

On Mon, May 20, 2019, 5:46 PM Louis Zipes <Louis.Zipes@kdrp.com wrote:

> Can you refresh me on how to capture the stack dumps you suggested?
> It's been a while....
>
> >In Task Manager find the Tomcat process, right click on it and there is
> an option to create a dump file.
>

Well, that will create Windows memory dump.

>
> >Also, if you have access to jconsole or other JMX tool you can connect
> to it and see what is going on.
>

Agreed. Alternatively you can use jstack to generate thread dumps. Exact
command will be like: jstack -l PID_of_JVM

I'll suggest to take thread dumps 1 sec apart.

But still I am surprised to see that it happens only during entire server
reboot and from your log, subsequent TC restart is not taking long time. Is
that a right statement or I am entirely misreading it? If that is true, you
can try 'atomatic delayed start' option.

>
> -----Original Message-----
> From: André Warnier (tomcat) [mailto:aw@ice-sa.com]
> Sent: Monday, May 20, 2019 4:44 AM
> To: users@tomcat.apache.org
> Subject: Re: Massive Startup Time after Server Reboot
>
> - - - external message, proceed with caution - - -
>
>
> On 20.05.2019 00:09, Jerry Malcolm wrote:
> > Just an FYI.... this server has been in production a little over a
> year.  This TC load
> > problem only started 2-3 months ago.  So 'something' changed to cause
> this problem to
> > manifest itself. It's been too long to try to correlate a specific
> server change to the
> > start of this problem. But I just wanted to point out that this problem
> has NOT been
> > around since day 1 on this server.
>
> Just in case, since this is a Windows server ..
> 2-3 months ago, an update of a virus scanner ?
> (of the intrusive kind which pre-checks each file that wants to be opened)
> Checking may be easy or not, depending on your access : disable the virus
> scanner just the
> time to start tomcat and check.
>
> >
> > Jerry
> >
> > On 5/19/2019 5:03 PM, Jerry Malcolm wrote:
> >> Rainer,
> >>
> >> No change with the urandom parm.  I am attaching a portion of the
> Catalina log.  The
> >> first half shows between 8 and 15 seconds to deploy each app for a
> single virtual host
> >> (there are no war files, the app is already exploded in the appbase
> dir).  I have quite
> >> a few virtual hosts with several apps each.  That initial server start
> took 21+ minutes.
> >>   I then restarted the TC service and got the ~.5 sec start per webapp
> shown in the 2nd
> >> half of the log below.
> >>
> >> Can you refresh me on how to capture the stack dumps you suggested?
> It's been a while....
> >>
> >> Thx.
> >>
> >> Jerry
> >>
> >>  First start of Tomcat after server reboot
> >> ---------------------------------------------------------------------
> >>  [C:\domains\<myDomain>.com\webapps\JSPWiki.war] has finished in
> [8,579] ms
> >> tory [C:\domains\<myDomain>.com\webapps\cis]
> >> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in
> [11,486] ms
> >> tory [C:\domains\<myDomain>.com\webapps\gallery]
> >> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in
> [9,204] ms
> >> tory [C:\domains\<myDomain>.com\webapps\gl]
> >> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in [8,469]
> ms
> >> tory [C:\domains\<myDomain>.com\webapps\idmanager]
> >> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
> [8,689] ms
> >> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
> >> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
> [6,907] ms
> >> tory [C:\domains\<myDomain>.com\webapps\malcolment]
> >> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished in
> [8,469] ms
> >> tory [C:\domains\<myDomain>.com\webapps\notebook]
> >> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in
> [10,189] ms
> >> tory [C:\domains\<myDomain>.com\webapps\order]
> >> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in
> [8,501] ms
> >> tory [C:\domains\<myDomain>.com\webapps\payment]
> >> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in
> [14,209] ms
> >> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
> >> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has
> finished in [9,018] ms
> >>  [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml]
> >> ptor [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml] has
> finished in [62] ms
> >>
> >>  Restart of Tomcat
> ---------------------------------------------------------------------
> >> tory [C:\domains\<myDomain>.com\webapps\cis]
> >> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in [594]
> ms
> >> tory [C:\domains\<myDomain>.com\webapps\gallery]
> >> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in
> [547] ms
> >> tory [C:\domains\<myDomain>.com\webapps\gl]
> >> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in [562] ms
> >> tory [C:\domains\<myDomain>.com\webapps\idmanager]
> >> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
> [578] ms
> >> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
> >> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
> [547] ms
> >> tory [C:\domains\<myDomain>.com\webapps\malcolment]
> >> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished in
> [579] ms
> >> tory [C:\domains\<myDomain>.com\webapps\notebook]
> >> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in
> [547] ms
> >> tory [C:\domains\<myDomain>.com\webapps\order]
> >> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in
> [531] ms
> >> tory [C:\domains\<myDomain>.com\webapps\payment]
> >> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in
> [562] ms
> >> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
> >> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has
> finished in [531] ms
> >>
> >> On 5/18/2019 3:18 PM, Rainer Jung wrote:
> >>> Most likely it hangs waiting for enough entropy for random number
> generator seeding.
> >>>
> >>> Try whether the problem goes away if you add
> >>>
> >>>   -Djava.security.egd=file:/dev/urandom
> >>>
> >>> to you process flags. If you are using older Java than Java 8 (not
> possible for Tomcat
> >>> 9 but just in case you also have older software stacks running), then
> it would be
> >>>
> >>>   -Djava.security.egd=file:/dev/./urandom
> >>>
> >>> The cryptic /./ is not a typo, you would need it for Java before Java
> 8.
> >>>
> >>> If this doesn't help, then you should try capturing a few stack dumps
> (thread dumps)
> >>> during the long startup time. People here can help interpret them.
> >>>
> >>> Regards,
> >>>
> >>> Rainer
> >>>
> >>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
> >>>> This is a weird one.  It started a few months ago.   I have TC 9
> running on Windows
> >>>> Server 16.  After I reboot the entire server, Tomcat takes forever on
> startup. It
> >>>> normally starts in about 30 seconds.  But after a server reboot it
> takes up to 15
> >>>> minutes... chugging along at a snail's pace starting up all of the
> apps on all of the
> >>>> virtual hosts.  It always finally gets there with everything
> successfully running.
> >>>> Other servers on the same box (Apache, JAMES, ISC BIND, MySQL) don't
> have any problem
> >>>> starting up. CPU, Disk, Memory, etc. usages are barely showing on the
> performance
> >>>> graphs. There's nothing in the Catalina log or system.err other than
> showing a couple
> >>>> of minute gap in the time stamp between each app as it starts up.  If
> I need to reboot
> >>>> again later, it boots again in about 30 sec as expected.
> >>>>
> >>>> First question... any ideas off the top of your head that might be
> causing this?
> >>>>
> >>>> Second question... is there any other logging I can turn on that
> might explain what TC
> >>>> is blocking on?
> >>>>
> >>>> Thanks.
> >>>>
> >>>> Jerry
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> ---------------------------------------
> CONFIDENTIALITY NOTICE: This message is for intended addressee(s) only and
> may contain information that is confidential, proprietary or exempt from
> disclosure. If you are not the intended recipient, please contact the
> sender immediately. Unauthorized use or distribution is prohibited and may
> be unlawful.
>