You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@openmeetings.apache.org by "Coscend@OM" <OM...@Coscend.com> on 2019/11/22 06:42:17 UTC
RE: OM5: Tomcat not shutting down due to OM5 method
Dear Maxim,
I am sending this to users, but we think one possibility (not confirmation) is this error is related to OM5 code [1]. There is a method in OM5 that is triggering memory leak while shutting down Tomcat. The most common reason is something hanging or delaying in Web application initializations [2]. This prevents Tomcat to properly shutdown [1, 2].
Or, is due to something else? (We are using OM5 binary from OM Web site. There are no customization or configuration changes.)
Here is the relevant log.
The key line in the log is: “The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak.”
------------------------
21-Nov-2019 23:24:30.024 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
21-Nov-2019 23:24:30.025 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
[1] https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005
[2] https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly…
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
From: Maxim Solodovnik [mailto:solomax666@gmail.com]
Sent: Friday, November 22, 2019 11:36 AM
To: dev <de...@openmeetings.apache.org>; OM.Insights@coscend.com
Subject: Re: OM5: Tomcat not shutting down due to OM5 method
Please send these messages to users@
It is not about OM/OM_community development
shutdown works for me as expected
please provide additional details (if any)
On Fri, 22 Nov 2019 at 13:04, Coscend@OM <OM.Insights@coscend.com <ma...@coscend.com> > wrote:
Dear OM Community,
While shutting down OM5-Tomcat via shutdown.sh, Tomcat is not shutting down.
These links state some method in Webapp could be preventing the shutdown
process of Tomcat:
https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shu <https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005>
tdown-port-8005
https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listen <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1>
ing?noredirect=1
<https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-liste <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1&lq=1>
ning?noredirect=1&lq=1> &lq=1
Could anyone provide any guidance what we need to modify in OM5 code or any
other place to properly shutdown Tomcat?
Here is the log snippet. There are several more such errors while shutting
down.
------------------------------
21-Nov-2019 23:24:29.843 INFO [main]
org.apache.catalina.core.StandardServer.await A valid shutdown command was
received via the shutdown port. Stopping the Server instance.
21-Nov-2019 23:24:29.844 INFO [main]
org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
["http-nio-5080"]
21-Nov-2019 23:24:29.850 INFO [main]
org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
["ajp-nio-8009"]
21-Nov-2019 23:24:29.856 INFO [main]
org.apache.catalina.core.StandardService.stopInternal Stopping service
[Catalina]
[34mINFO [0;39m 11-21 23:24:29.884 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket extensions initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket core library initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy:
org.wicketstuff.dashboard.DashboardContextInitializer@72690a1 <ma...@72690a1>
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
(theme-default)
21-Nov-2019 23:24:29.886 INFO [main]
com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
[172.17.0.1]:5701 is SHUTTING_DOWN
21-Nov-2019 23:24:29.891 INFO [hz.server-1.cached.thread-2]
com.hazelcast.internal.partition.impl.MigrationManager.null
[172.17.0.1]:5701 [dev] [3.12] Shutdown request of Member [172.17.0.1]:5701
- 799babea-5e78-4cf9-a099-f4b1fb09b9c9 this is handled
21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Shutting down connection manager...
21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Shutting down node engine...
21-Nov-2019 23:24:29.979 INFO [main]
com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12]
Destroying node NodeExtension.
21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Hazelcast Shutdown is completed in 90 ms.
21-Nov-2019 23:24:29.980 INFO [main]
com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
[172.17.0.1]:5701 is SHUTDOWN
21-Nov-2019 23:24:30.023 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web
application [Openmeetings] registered the JDBC driver
[com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
application was stopped. To prevent a memory leak, the JDBC Driver has been
forcibly unregistered.
21-Nov-2019 23:24:30.024 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [Openmeetings] appears to have started a thread named
[nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThr
eadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventEx
ecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable
.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
21-Nov-2019 23:24:30.025 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [Openmeetings] appears to have started a thread named
[org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
has failed to stop it. This is very likely to create a memory leak. Stack
trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555
)
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com <http://www.Coscend.com>
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education,
Telepresence Services, on the fly.
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
Messages from Coscend Communications Solutions' posted at:
<http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
--
WBR
Maxim aka solomax
Re: OM5: Tomcat not shutting down due to OM5 method
Posted by Maxim Solodovnik <so...@gmail.com>.
Glad to hear it
What was the problem?
On Mon, 9 Dec 2019 at 16:44, Coscend@OM <OM...@coscend.com> wrote:
> Dear Maxim,
>
>
>
> Thank you for the pointers. These hints directed us to a resolution.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> www.Coscend.com <http://www.coscend.com/>
>
> ------------------------------------------------------------------
>
> *Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education, Telepresence Services, on the fly…*
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
>
>
>
>
>
>
>
>
>
>
>
> *From:* Maxim Solodovnik [mailto:solomax666@gmail.com]
> *Sent:* Monday, December 9, 2019 10:28 AM
> *To:* Openmeetings user-list <us...@openmeetings.apache.org>;
> OM.Insights@coscend.com
> *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> conf/server.xml should have this line:
> https://github.com/apache/openmeetings/blob/master/openmeetings-server/src/main/assembly/conf/server.xml#L22
>
> to be able to perform shutdown
>
> You can check Tomcat documentation and contact users@tomcat.a.o mailing
> list :)
>
>
>
> On Sun, 8 Dec 2019 at 06:16, Coscend@OM <OM...@coscend.com> wrote:
>
> Dear Maxim,
>
>
>
> è The error "Could not contact [localhost:8005] " most probably means:
> you have disabled AJP. This is why shutdown.sh can do nothing
>
>
>
> Our server.xml setting:
>
> <Connector port="5443"
> protocol="org.apache.coyote.http11.Http11AprProtocol"
>
> maxThreads="150" SSLEnabled="true" >
>
> <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"
> />
>
> <SSLHostConfig>
>
> <Certificate certificateKeyFile="/etc/….key"
>
> certificateFile="/etc/...cer"
>
> type="EC" />
>
> </SSLHostConfig>
>
> </Connector>
>
> …
>
> <Connector port="8009" protocol="AJP/1.3" redirectPort="<5443>" />
>
>
>
> Question: There is a line in the log. After this line, there are several
> errors.
>
>
>
> 07-Dec-2019 16:55:45.082 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
> web application [Openmeetings] registered the JDBC driver
> [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
> application was stopped. To prevent a memory leak, the JDBC Driver has been
> forcibly unregistered.
>
>
>
>
>
> ----------
>
> Here is the OM log:
>
>
>
> 07-Dec-2019 16:55:44.938 INFO [main]
> org.apache.catalina.core.StandardServer.await A valid shutdown command was
> received via the shutdown port. Stopping the Server instance.
>
> 07-Dec-2019 16:55:44.938 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["https-openssl-apr-5443"]
>
> 07-Dec-2019 16:55:44.945 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["ajp-nio-8009"]
>
> 07-Dec-2019 16:55:44.950 INFO [main]
> org.apache.catalina.core.StandardService.stopInternal Stopping service
> [Catalina]
>
> [34mINFO [0;39m 12-07 16:55:44.983 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy:
> org.wicketstuff.dashboard.DashboardContextInitializer@10c541dc
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
> (theme-default)
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket core library initializer
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket extensions initializer
>
> 07-Dec-2019 16:55:44.984 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12.2]
> [172.17.0.1]:5701 is SHUTTING_DOWN
>
> 07-Dec-2019 16:55:44.990 INFO [hz.server-1.cached.thread-4]
> com.hazelcast.internal.partition.impl.MigrationManager.null
> [172.17.0.1]:5701 [dev] [3.12.2] Shutdown request of Member
> [172.17.0.1]:5701 - 7d3e2187-936e-49b1-a0b7-cdb0f2a604d6 this is handled
>
> 07-Dec-2019 16:55:44.996 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12.2] Shutting down connection manager...
>
> 07-Dec-2019 16:55:44.997 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12.2] Shutting down node engine...
>
> 07-Dec-2019 16:55:45.011 INFO [main]
> com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12.2]
> Destroying node NodeExtension.
>
> 07-Dec-2019 16:55:45.011 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12.2] Hazelcast Shutdown is completed in 22 ms.
>
> 07-Dec-2019 16:55:45.011 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12.2]
> [172.17.0.1]:5701 is SHUTDOWN
>
> 07-Dec-2019 16:55:45.082 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
> web application [Openmeetings] registered the JDBC driver
> [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
> application was stopped. To prevent a memory leak, the JDBC Driver has been
> forcibly unregistered.
>
> 07-Dec-2019 16:55:45.083 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:720)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:510)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
>
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 07-Dec-2019 16:55:45.083 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> www.Coscend.com <http://www.coscend.com/>
>
> ------------------------------------------------------------------
>
> *Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education, Telepresence Services, on the fly…*
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
>
>
>
>
>
>
>
> *From:* Maxim Solodovnik [mailto:solomax666@gmail.com]
> *Sent:* Tuesday, November 26, 2019 9:00 PM
> *To:* Openmeetings user-list <us...@openmeetings.apache.org>;
> OM.Insights@coscend.com
> *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Hello Hemant,
>
>
>
> Vanilla OM5 doen't have MySQL driver
>
> It also not uses port 443
>
>
>
> I would suggest to test OM with no modifications (before ANY
> customizations)
>
>
>
> The error "Could not contact [localhost:8005] " most probably means: you
> have disabled AJP
>
> This is why shutdown.sh can do nothing
>
> you can still use `kill -9` in this case
>
>
>
> On Tue, 26 Nov 2019 at 21:59, Coscend@OM <OM...@coscend.com> wrote:
>
> Dear Maxim,
>
>
>
> Vanilla Tomcat9: startup and shutdown fine.
>
>
>
> Vanilla OM5
>
> User: when the binary is unarchived, it has a local usergroup as the
> default owner. Is this correct?
>
>
>
> (binary from OpenMeetings Web site): no configuration changes except SSL
> certificate input.
>
>
>
>
>
> Startup fine
>
> Shutdown:
>
> netstat gives no port binding on 443 and no pid
>
> echo $CATALINA_PID gives blank
>
> BUT
>
> ps aux gives tomcat pid running
>
>
>
> Below are CLI log and OM5 / Catalina log.
>
>
>
>
>
> If we run shutdown command again, CLI gives “Tomcat may not be running”:
>
>
>
> -------------------------------------------------------
>
> CLI log
>
>
>
> org.apache.catalina.startup.Catalina stopServer
>
> SEVERE: Could not contact [localhost:8005] (base port [8005] and offset
> [0]). Tomcat may not be running.
>
> Nov 26, 2019 8:39:11 AM org.apache.catalina.startup.Catalina stopServer
>
> SEVERE: Error stopping Catalina
>
> java.net.ConnectException: Connection refused (Connection refused)
>
> at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
>
> at
> java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
>
> at
> java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
>
> at
> java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
>
> at java.base/java.net.Socket.connect(Socket.java:591)
>
> at java.base/java.net.Socket.connect(Socket.java:540)
>
> at java.base/java.net.Socket.<init>(Socket.java:436)
>
> at java.base/java.net.Socket.<init>(Socket.java:213)
>
> at
> org.apache.catalina.startup.Catalina.stopServer(Catalina.java:513)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:390)
>
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:480)
>
>
>
>
>
> OM5 log
>
>
>
> 26-Nov-2019 08:25:49.733 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
> [172.17.0.1]:5701 is SHUTDOWN
>
> 26-Nov-2019 08:25:49.790 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
> web application [Openmeetings] registered the JDBC driver
> [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
> application was stopped. To prevent a memory leak, the JDBC Driver has been
> forcibly unregistered.
>
> 26-Nov-2019 08:25:49.791 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
> 26-Nov-2019 08:25:49.792 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
> 26-Nov-2019 08:25:49.793 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10]
> but has failed to stop it. This is very likely to create a memory leak.
> Stack trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
> 26-Nov-2019 08:25:49.793 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [mysql-cj-abandoned-connection-cleanup] but has failed to stop it. This is
> very likely to create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
> java.base@11/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
>
>
> com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:85)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.794 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [pool-5-thread-1] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
>
> java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
>
>
> java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
>
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.795 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [pool-1-thread-1] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
>
> java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
>
>
> java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
>
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.795 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.796 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [pool-21-thread-1] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
>
> java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
>
>
> java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
>
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.796 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-2] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.797 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-3] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
>
>
> …..REPEATS SEVERAL TIMES
>
>
>
> 26-Nov-2019 08:25:49.814 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-30] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.815 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@560478b8]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.816 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a
> value of type [org.apache.wicket.ThreadContext] (value
> [org.apache.wicket.ThreadContext@12b21dea]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.816 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a
> value of type [org.apache.wicket.ThreadContext] (value
> [org.apache.wicket.ThreadContext@e41cc5d]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak…
>
>
>
> …REPEATS SEVERAL TIMES
>
>
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a
> value of type [org.apache.wicket.ThreadContext] (value
> [org.apache.wicket.ThreadContext@4e2fb8d6]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@1c5e617f]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@5147f2c6]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@ca5a0d3]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.823 INFO [main]
> org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler
> ["https-openssl-apr-443"]
>
> 26-Nov-2019 08:25:49.874 INFO [main]
> org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler
> ["ajp-nio-8009"]
>
> 26-Nov-2019 08:25:49.879 INFO [main]
> org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler
> ["https-openssl-apr-443"]
>
> 26-Nov-2019 08:25:49.881 INFO [main]
> org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler
> ["ajp-nio-8009"]
>
> 26-Nov-2019 08:25:49.913 INFO [mysql-cj-abandoned-connection-cleanup]
> org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading
> Illegal access: this web application instance has been stopped already.
> Could not load []. The following stack trace is thrown for debugging
> purposes as well as to attempt to terminate the thread which caused the
> illegal access.
>
> java.lang.IllegalStateException: Illegal access: this
> web application instance has been stopped already. Could not load []. The
> following stack trace is thrown for debugging purposes as well as to
> attempt to terminate the thread which caused the illegal access.
>
> at
> org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1385)
>
> at
> org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1038)
>
> at
> com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkThreadContextClassLoader(AbandonedConnectionCleanupThread.java:117)
>
> at
> com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:84)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at
> java.base/java.lang.Thread.run(Thread.java:834)
>
>
>
>
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> www.Coscend.com <http://www.coscend.com/>
>
> ------------------------------------------------------------------
>
> *Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education, Telepresence Services, on the fly…*
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
>
>
>
>
>
>
>
>
>
> *From:* Maxim Solodovnik [mailto:solomax666@gmail.com]
> *Sent:* Monday, November 25, 2019 10:21 AM
> *To:* Openmeetings user-list <us...@openmeetings.apache.org>;
> OM.Insights@coscend.com
> *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Hello Hemant,
>
>
>
> dev@ mailing list is for discussing OM/Om community development (i.e.
> discussing patch, the way to solve something, etc.)
>
> NOT for contacting OM devs to report an issue
>
>
>
> I don't have CentOS8 so I need detailed description of
>
> what do you mean by "Environment is CentOS 8 (using docker) and MariaDB
> 10.3.x"? (OM is java application, it doesn't require docker to be able to
> run)
>
>
>
> So could you run "vanilla" OM5 (without any other services, proxies, 3rd
> parties etc.) and check it it works for you as expected? i.e. both
> startup.sh and shutdown.sh run as correct user do it's job?
>
>
>
> If yes - add changes one by one and continue checking
>
> If not - compare with "vanilla" Tomcat9 - what is different :)
>
> baby steps :)
>
>
>
> On Fri, 22 Nov 2019 at 13:49, Coscend@OM <OM...@coscend.com> wrote:
>
> Further, if we do not run OM5, Tomcat shuts down properly. Environment is
> CentOS 8 (using docker) and MariaDB 10.3.x
>
>
>
> *From:* Coscend@OM [mailto:OM.Insights@Coscend.com]
> *Sent:* Friday, November 22, 2019 12:12 PM
> *To:* user@openmeetings.apache.org
> *Subject:* RE: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Dear Maxim,
>
>
>
> I am sending this to users, but we think one possibility (not
> confirmation) is this error is related to OM5 code [1]. There is a method
> in OM5 that is triggering memory leak while shutting down Tomcat. The most
> common reason is something hanging or delaying in Web application
> initializations [2]. This prevents Tomcat to properly shutdown [1, 2].
>
>
>
> Or, is due to something else? (We are using OM5 binary from OM Web site.
> There are no customization or configuration changes.)
>
>
>
> Here is the relevant log.
>
>
>
> The key line in the log is: “The web application [Openmeetings] appears
> to have started a thread named [nioEventLoopGroup-2-1] but has failed to
> stop it. This is very likely to create a memory leak.”
>
> ------------------------
>
> 21-Nov-2019 23:24:30.024 WARNING [main]
>
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
>
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
>
>
> 21-Nov-2019 23:24:30.025 WARNING [main]
>
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:java.base@11/java.lang.Object.wait(Native Method)
>
>
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
>
>
>
>
> [1]
> https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005
>
>
>
> [2]
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1
>
>
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> www.Coscend.com <http://www.coscend.com/>
>
> ------------------------------------------------------------------
>
> *Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education, Telepresence Services, on the fly…*
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
>
>
>
>
>
>
>
> *From:* Maxim Solodovnik [mailto:solomax666@gmail.com
> <so...@gmail.com>]
> *Sent:* Friday, November 22, 2019 11:36 AM
> *To:* dev <de...@openmeetings.apache.org>; OM.Insights@coscend.com
> *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Please send these messages to users@
>
> It is not about OM/OM_community development
>
>
>
> shutdown works for me as expected
>
> please provide additional details (if any)
>
>
>
> On Fri, 22 Nov 2019 at 13:04, Coscend@OM <OM...@coscend.com> wrote:
>
> Dear OM Community,
>
>
>
> While shutting down OM5-Tomcat via shutdown.sh, Tomcat is not shutting
> down.
> These links state some method in Webapp could be preventing the shutdown
> process of Tomcat:
>
>
> https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shu
> tdown-port-8005
>
>
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listen
> ing?noredirect=1
> <
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-liste
> ning?noredirect=1&lq=1> &lq=1
>
>
>
> Could anyone provide any guidance what we need to modify in OM5 code or any
> other place to properly shutdown Tomcat?
>
>
>
> Here is the log snippet. There are several more such errors while shutting
> down.
>
>
>
> ------------------------------
>
>
>
> 21-Nov-2019 23:24:29.843 INFO [main]
> org.apache.catalina.core.StandardServer.await A valid shutdown command was
> received via the shutdown port. Stopping the Server instance.
>
> 21-Nov-2019 23:24:29.844 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["http-nio-5080"]
>
> 21-Nov-2019 23:24:29.850 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["ajp-nio-8009"]
>
> 21-Nov-2019 23:24:29.856 INFO [main]
> org.apache.catalina.core.StandardService.stopInternal Stopping service
> [Catalina]
>
> [34mINFO [0;39m 11-21 23:24:29.884 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket extensions initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket core library initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy:
> org.wicketstuff.dashboard.DashboardContextInitializer@72690a1
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
> (theme-default)
>
> 21-Nov-2019 23:24:29.886 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
> [172.17.0.1]:5701 is SHUTTING_DOWN
>
> 21-Nov-2019 23:24:29.891 INFO [hz.server-1.cached.thread-2]
> com.hazelcast.internal.partition.impl.MigrationManager.null
> [172.17.0.1]:5701 [dev] [3.12] Shutdown request of Member [172.17.0.1]:5701
> - 799babea-5e78-4cf9-a099-f4b1fb09b9c9 this is handled
>
> 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Shutting down connection manager...
>
> 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Shutting down node engine...
>
> 21-Nov-2019 23:24:29.979 INFO [main]
> com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12]
> Destroying node NodeExtension.
>
> 21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Hazelcast Shutdown is completed in 90 ms.
>
> 21-Nov-2019 23:24:29.980 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
> [172.17.0.1]:5701 is SHUTDOWN
>
> 21-Nov-2019 23:24:30.023 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
> web
> application [Openmeetings] registered the JDBC driver
> [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
> application was stopped. To prevent a memory leak, the JDBC Driver has been
> forcibly unregistered.
>
> 21-Nov-2019 23:24:30.024 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThr
> eadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventEx
> ecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable
> .java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 21-Nov-2019 23:24:30.025 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555
> )
>
>
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> <http://www.coscend.com/> www.Coscend.com
>
> ------------------------------------------------------------------
>
> Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education,
> Telepresence Services, on the fly.
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
>
>
>
>
> --
>
> WBR
> Maxim aka solomax
>
>
>
>
> --
>
> WBR
> Maxim aka solomax
>
>
>
>
> --
>
> WBR
> Maxim aka solomax
>
>
>
>
> --
>
> WBR
> Maxim aka solomax
>
--
WBR
Maxim aka solomax
RE: OM5: Tomcat not shutting down due to OM5 method
Posted by "Coscend@OM" <OM...@Coscend.com>.
Dear Maxim,
Thank you for the pointers. These hints directed us to a resolution.
Sincerely,
Hemant K. Sabat
www.Coscend.com <http://www.coscend.com/>
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly…
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
From: Maxim Solodovnik [mailto:solomax666@gmail.com]
Sent: Monday, December 9, 2019 10:28 AM
To: Openmeetings user-list <us...@openmeetings.apache.org>; OM.Insights@coscend.com
Subject: Re: OM5: Tomcat not shutting down due to OM5 method
conf/server.xml should have this line: https://github.com/apache/openmeetings/blob/master/openmeetings-server/src/main/assembly/conf/server.xml#L22
to be able to perform shutdown
You can check Tomcat documentation and contact users@tomcat.a.o <ma...@tomcat.a.o> mailing list :)
On Sun, 8 Dec 2019 at 06:16, Coscend@OM <OM.Insights@coscend.com <ma...@coscend.com> > wrote:
Dear Maxim,
è The error "Could not contact [localhost:8005] " most probably means: you have disabled AJP. This is why shutdown.sh can do nothing
Our server.xml setting:
<Connector port="5443" protocol="org.apache.coyote.http11.Http11AprProtocol"
maxThreads="150" SSLEnabled="true" >
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
<SSLHostConfig>
<Certificate certificateKeyFile="/etc/….key"
certificateFile="/etc/...cer"
type="EC" />
</SSLHostConfig>
</Connector>
…
<Connector port="8009" protocol="AJP/1.3" redirectPort="<5443>" />
Question: There is a line in the log. After this line, there are several errors.
07-Dec-2019 16:55:45.082 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [Openmeetings] registered the JDBC driver [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
----------
Here is the OM log:
07-Dec-2019 16:55:44.938 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
07-Dec-2019 16:55:44.938 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-openssl-apr-5443"]
07-Dec-2019 16:55:44.945 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-8009"]
07-Dec-2019 16:55:44.950 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
[34mINFO [0;39m 12-07 16:55:44.983 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: org.wicketstuff.dashboard.DashboardContextInitializer@10c541dc <ma...@10c541dc>
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer (theme-default)
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket core library initializer
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket extensions initializer
07-Dec-2019 16:55:44.984 INFO [main] com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12.2] [172.17.0.1]:5701 is SHUTTING_DOWN
07-Dec-2019 16:55:44.990 INFO [hz.server-1.cached.thread-4] com.hazelcast.internal.partition.impl.MigrationManager.null [172.17.0.1]:5701 [dev] [3.12.2] Shutdown request of Member [172.17.0.1]:5701 - 7d3e2187-936e-49b1-a0b7-cdb0f2a604d6 this is handled
07-Dec-2019 16:55:44.996 INFO [main] com.hazelcast.instance.Node.null [172.17.0.1]:5701 [dev] [3.12.2] Shutting down connection manager...
07-Dec-2019 16:55:44.997 INFO [main] com.hazelcast.instance.Node.null [172.17.0.1]:5701 [dev] [3.12.2] Shutting down node engine...
07-Dec-2019 16:55:45.011 INFO [main] com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12.2] Destroying node NodeExtension.
07-Dec-2019 16:55:45.011 INFO [main] com.hazelcast.instance.Node.null [172.17.0.1]:5701 [dev] [3.12.2] Hazelcast Shutdown is completed in 22 ms.
07-Dec-2019 16:55:45.011 INFO [main] com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12.2] [172.17.0.1]:5701 is SHUTDOWN
07-Dec-2019 16:55:45.082 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [Openmeetings] registered the JDBC driver [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
07-Dec-2019 16:55:45.083 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:720)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:510)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
07-Dec-2019 16:55:45.083 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly…
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
From: Maxim Solodovnik [mailto:solomax666@gmail.com <ma...@gmail.com> ]
Sent: Tuesday, November 26, 2019 9:00 PM
To: Openmeetings user-list <user@openmeetings.apache.org <ma...@openmeetings.apache.org> >; OM.Insights@coscend.com <ma...@coscend.com>
Subject: Re: OM5: Tomcat not shutting down due to OM5 method
Hello Hemant,
Vanilla OM5 doen't have MySQL driver
It also not uses port 443
I would suggest to test OM with no modifications (before ANY customizations)
The error "Could not contact [localhost:8005] " most probably means: you have disabled AJP
This is why shutdown.sh can do nothing
you can still use `kill -9` in this case
On Tue, 26 Nov 2019 at 21:59, Coscend@OM <OM.Insights@coscend.com <ma...@coscend.com> > wrote:
Dear Maxim,
Vanilla Tomcat9: startup and shutdown fine.
Vanilla OM5
User: when the binary is unarchived, it has a local usergroup as the default owner. Is this correct?
(binary from OpenMeetings Web site): no configuration changes except SSL certificate input.
Startup fine
Shutdown:
netstat gives no port binding on 443 and no pid
echo $CATALINA_PID gives blank
BUT
ps aux gives tomcat pid running
Below are CLI log and OM5 / Catalina log.
If we run shutdown command again, CLI gives “Tomcat may not be running”:
-------------------------------------------------------
CLI log
org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact [localhost:8005] (base port [8005] and offset [0]). Tomcat may not be running.
Nov 26, 2019 8:39:11 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Error stopping Catalina
java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at java.base/java.net.Socket.connect(Socket.java:540)
at java.base/java.net.Socket.<init>(Socket.java:436)
at java.base/java.net.Socket.<init>(Socket.java:213)
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:513)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:390)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:480)
OM5 log
26-Nov-2019 08:25:49.733 INFO [main] com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12] [172.17.0.1]:5701 is SHUTDOWN
26-Nov-2019 08:25:49.790 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [Openmeetings] registered the JDBC driver [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
26-Nov-2019 08:25:49.791 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
26-Nov-2019 08:25:49.792 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
26-Nov-2019 08:25:49.793 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
26-Nov-2019 08:25:49.793 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [mysql-cj-abandoned-connection-cleanup] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
java.base@11/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155) <mailto:java.base@11/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)>
com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:85)
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)>
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.794 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/jdk.internal.misc.Unsafe.park(Native <mailto:java.base@11/jdk.internal.misc.Unsafe.park(Native> Method)
java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) <mailto:java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)>
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) <mailto:java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)>
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.795 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/jdk.internal.misc.Unsafe.park(Native <mailto:java.base@11/jdk.internal.misc.Unsafe.park(Native> Method)
java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) <mailto:java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)>
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) <mailto:java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)>
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.795 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.796 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [pool-21-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/jdk.internal.misc.Unsafe.park(Native <mailto:java.base@11/jdk.internal.misc.Unsafe.park(Native> Method)
java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) <mailto:java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)>
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) <mailto:java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)>
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.796 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.797 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
…..REPEATS SEVERAL TIMES
26-Nov-2019 08:25:49.814 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-30] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.815 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@560478b8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.816 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a value of type [org.apache.wicket.ThreadContext] (value [org.apache.wicket.ThreadContext@12b21dea]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.816 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a value of type [org.apache.wicket.ThreadContext] (value [org.apache.wicket.ThreadContext@e41cc5d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak…
…REPEATS SEVERAL TIMES
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a value of type [org.apache.wicket.ThreadContext] (value [org.apache.wicket.ThreadContext@4e2fb8d6]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1c5e617f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@5147f2c6]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@ca5a0d3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.823 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-openssl-apr-443"]
26-Nov-2019 08:25:49.874 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-8009"]
26-Nov-2019 08:25:49.879 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-openssl-apr-443"]
26-Nov-2019 08:25:49.881 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-8009"]
26-Nov-2019 08:25:49.913 INFO [mysql-cj-abandoned-connection-cleanup] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load []. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load []. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1385)
at org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1038)
at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkThreadContextClassLoader(AbandonedConnectionCleanupThread.java:117)
at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:84)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Thank you.
Sincerely,
Hemant K. Sabat
www.Coscend.com <http://www.coscend.com/>
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly…
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
From: Maxim Solodovnik [mailto:solomax666@gmail.com <ma...@gmail.com> ]
Sent: Monday, November 25, 2019 10:21 AM
To: Openmeetings user-list <user@openmeetings.apache.org <ma...@openmeetings.apache.org> >; OM.Insights@coscend.com <ma...@coscend.com>
Subject: Re: OM5: Tomcat not shutting down due to OM5 method
Hello Hemant,
dev@ mailing list is for discussing OM/Om community development (i.e. discussing patch, the way to solve something, etc.)
NOT for contacting OM devs to report an issue
I don't have CentOS8 so I need detailed description of
what do you mean by "Environment is CentOS 8 (using docker) and MariaDB 10.3.x"? (OM is java application, it doesn't require docker to be able to run)
So could you run "vanilla" OM5 (without any other services, proxies, 3rd parties etc.) and check it it works for you as expected? i.e. both startup.sh and shutdown.sh run as correct user do it's job?
If yes - add changes one by one and continue checking
If not - compare with "vanilla" Tomcat9 - what is different :)
baby steps :)
On Fri, 22 Nov 2019 at 13:49, Coscend@OM <OM.Insights@coscend.com <ma...@coscend.com> > wrote:
Further, if we do not run OM5, Tomcat shuts down properly. Environment is CentOS 8 (using docker) and MariaDB 10.3.x
From: Coscend@OM [mailto:OM.Insights@Coscend.com <ma...@Coscend.com> ]
Sent: Friday, November 22, 2019 12:12 PM
To: user@openmeetings.apache.org <ma...@openmeetings.apache.org>
Subject: RE: OM5: Tomcat not shutting down due to OM5 method
Dear Maxim,
I am sending this to users, but we think one possibility (not confirmation) is this error is related to OM5 code [1]. There is a method in OM5 that is triggering memory leak while shutting down Tomcat. The most common reason is something hanging or delaying in Web application initializations [2]. This prevents Tomcat to properly shutdown [1, 2].
Or, is due to something else? (We are using OM5 binary from OM Web site. There are no customization or configuration changes.)
Here is the relevant log.
The key line in the log is: “The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak.”
------------------------
21-Nov-2019 23:24:30.024 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
21-Nov-2019 23:24:30.025 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
[1] https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005
[2] https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly…
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
From: Maxim Solodovnik [mailto:solomax666@gmail.com]
Sent: Friday, November 22, 2019 11:36 AM
To: dev <dev@openmeetings.apache.org <ma...@openmeetings.apache.org> >; OM.Insights@coscend.com <ma...@coscend.com>
Subject: Re: OM5: Tomcat not shutting down due to OM5 method
Please send these messages to users@
It is not about OM/OM_community development
shutdown works for me as expected
please provide additional details (if any)
On Fri, 22 Nov 2019 at 13:04, Coscend@OM <OM.Insights@coscend.com <ma...@coscend.com> > wrote:
Dear OM Community,
While shutting down OM5-Tomcat via shutdown.sh, Tomcat is not shutting down.
These links state some method in Webapp could be preventing the shutdown
process of Tomcat:
https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shu <https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005>
tdown-port-8005
https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listen <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1>
ing?noredirect=1
<https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-liste <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1&lq=1>
ning?noredirect=1&lq=1> &lq=1
Could anyone provide any guidance what we need to modify in OM5 code or any
other place to properly shutdown Tomcat?
Here is the log snippet. There are several more such errors while shutting
down.
------------------------------
21-Nov-2019 23:24:29.843 INFO [main]
org.apache.catalina.core.StandardServer.await A valid shutdown command was
received via the shutdown port. Stopping the Server instance.
21-Nov-2019 23:24:29.844 INFO [main]
org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
["http-nio-5080"]
21-Nov-2019 23:24:29.850 INFO [main]
org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
["ajp-nio-8009"]
21-Nov-2019 23:24:29.856 INFO [main]
org.apache.catalina.core.StandardService.stopInternal Stopping service
[Catalina]
[34mINFO [0;39m 11-21 23:24:29.884 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket extensions initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket core library initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy:
org.wicketstuff.dashboard.DashboardContextInitializer@72690a1 <ma...@72690a1>
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
(theme-default)
21-Nov-2019 23:24:29.886 INFO [main]
com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
[172.17.0.1]:5701 is SHUTTING_DOWN
21-Nov-2019 23:24:29.891 INFO [hz.server-1.cached.thread-2]
com.hazelcast.internal.partition.impl.MigrationManager.null
[172.17.0.1]:5701 [dev] [3.12] Shutdown request of Member [172.17.0.1]:5701
- 799babea-5e78-4cf9-a099-f4b1fb09b9c9 this is handled
21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Shutting down connection manager...
21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Shutting down node engine...
21-Nov-2019 23:24:29.979 INFO [main]
com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12]
Destroying node NodeExtension.
21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Hazelcast Shutdown is completed in 90 ms.
21-Nov-2019 23:24:29.980 INFO [main]
com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
[172.17.0.1]:5701 is SHUTDOWN
21-Nov-2019 23:24:30.023 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web
application [Openmeetings] registered the JDBC driver
[com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
application was stopped. To prevent a memory leak, the JDBC Driver has been
forcibly unregistered.
21-Nov-2019 23:24:30.024 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [Openmeetings] appears to have started a thread named
[nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThr
eadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventEx
ecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable
.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
21-Nov-2019 23:24:30.025 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [Openmeetings] appears to have started a thread named
[org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
has failed to stop it. This is very likely to create a memory leak. Stack
trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555
)
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com <http://www.Coscend.com>
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education,
Telepresence Services, on the fly.
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
Messages from Coscend Communications Solutions' posted at:
<http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
--
WBR
Maxim aka solomax
--
WBR
Maxim aka solomax
--
WBR
Maxim aka solomax
--
WBR
Maxim aka solomax
Re: OM5: Tomcat not shutting down due to OM5 method
Posted by Maxim Solodovnik <so...@gmail.com>.
conf/server.xml should have this line:
https://github.com/apache/openmeetings/blob/master/openmeetings-server/src/main/assembly/conf/server.xml#L22
to be able to perform shutdown
You can check Tomcat documentation and contact users@tomcat.a.o mailing
list :)
On Sun, 8 Dec 2019 at 06:16, Coscend@OM <OM...@coscend.com> wrote:
> Dear Maxim,
>
>
>
> è The error "Could not contact [localhost:8005] " most probably means:
> you have disabled AJP. This is why shutdown.sh can do nothing
>
>
>
> Our server.xml setting:
>
> <Connector port="5443"
> protocol="org.apache.coyote.http11.Http11AprProtocol"
>
> maxThreads="150" SSLEnabled="true" >
>
> <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"
> />
>
> <SSLHostConfig>
>
> <Certificate certificateKeyFile="/etc/….key"
>
> certificateFile="/etc/...cer"
>
> type="EC" />
>
> </SSLHostConfig>
>
> </Connector>
>
> …
>
> <Connector port="8009" protocol="AJP/1.3" redirectPort="<5443>" />
>
>
>
> Question: There is a line in the log. After this line, there are several
> errors.
>
>
>
> 07-Dec-2019 16:55:45.082 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
> web application [Openmeetings] registered the JDBC driver
> [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
> application was stopped. To prevent a memory leak, the JDBC Driver has been
> forcibly unregistered.
>
>
>
>
>
> ----------
>
> Here is the OM log:
>
>
>
> 07-Dec-2019 16:55:44.938 INFO [main]
> org.apache.catalina.core.StandardServer.await A valid shutdown command was
> received via the shutdown port. Stopping the Server instance.
>
> 07-Dec-2019 16:55:44.938 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["https-openssl-apr-5443"]
>
> 07-Dec-2019 16:55:44.945 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["ajp-nio-8009"]
>
> 07-Dec-2019 16:55:44.950 INFO [main]
> org.apache.catalina.core.StandardService.stopInternal Stopping service
> [Catalina]
>
> [34mINFO [0;39m 12-07 16:55:44.983 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy:
> org.wicketstuff.dashboard.DashboardContextInitializer@10c541dc
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
> (theme-default)
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket core library initializer
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
>
> [34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m
> - [OpenmeetingsApplication] destroy: Wicket extensions initializer
>
> 07-Dec-2019 16:55:44.984 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12.2]
> [172.17.0.1]:5701 is SHUTTING_DOWN
>
> 07-Dec-2019 16:55:44.990 INFO [hz.server-1.cached.thread-4]
> com.hazelcast.internal.partition.impl.MigrationManager.null
> [172.17.0.1]:5701 [dev] [3.12.2] Shutdown request of Member
> [172.17.0.1]:5701 - 7d3e2187-936e-49b1-a0b7-cdb0f2a604d6 this is handled
>
> 07-Dec-2019 16:55:44.996 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12.2] Shutting down connection manager...
>
> 07-Dec-2019 16:55:44.997 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12.2] Shutting down node engine...
>
> 07-Dec-2019 16:55:45.011 INFO [main]
> com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12.2]
> Destroying node NodeExtension.
>
> 07-Dec-2019 16:55:45.011 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12.2] Hazelcast Shutdown is completed in 22 ms.
>
> 07-Dec-2019 16:55:45.011 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12.2]
> [172.17.0.1]:5701 is SHUTDOWN
>
> 07-Dec-2019 16:55:45.082 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
> web application [Openmeetings] registered the JDBC driver
> [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
> application was stopped. To prevent a memory leak, the JDBC Driver has been
> forcibly unregistered.
>
> 07-Dec-2019 16:55:45.083 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:720)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:510)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
>
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 07-Dec-2019 16:55:45.083 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> www.Coscend.com <http://www.coscend.com/>
>
> ------------------------------------------------------------------
>
> *Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education, Telepresence Services, on the fly…*
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
>
>
>
>
>
>
>
> *From:* Maxim Solodovnik [mailto:solomax666@gmail.com]
> *Sent:* Tuesday, November 26, 2019 9:00 PM
> *To:* Openmeetings user-list <us...@openmeetings.apache.org>;
> OM.Insights@coscend.com
> *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Hello Hemant,
>
>
>
> Vanilla OM5 doen't have MySQL driver
>
> It also not uses port 443
>
>
>
> I would suggest to test OM with no modifications (before ANY
> customizations)
>
>
>
> The error "Could not contact [localhost:8005] " most probably means: you
> have disabled AJP
>
> This is why shutdown.sh can do nothing
>
> you can still use `kill -9` in this case
>
>
>
> On Tue, 26 Nov 2019 at 21:59, Coscend@OM <OM...@coscend.com> wrote:
>
> Dear Maxim,
>
>
>
> Vanilla Tomcat9: startup and shutdown fine.
>
>
>
> Vanilla OM5
>
> User: when the binary is unarchived, it has a local usergroup as the
> default owner. Is this correct?
>
>
>
> (binary from OpenMeetings Web site): no configuration changes except SSL
> certificate input.
>
>
>
>
>
> Startup fine
>
> Shutdown:
>
> netstat gives no port binding on 443 and no pid
>
> echo $CATALINA_PID gives blank
>
> BUT
>
> ps aux gives tomcat pid running
>
>
>
> Below are CLI log and OM5 / Catalina log.
>
>
>
>
>
> If we run shutdown command again, CLI gives “Tomcat may not be running”:
>
>
>
> -------------------------------------------------------
>
> CLI log
>
>
>
> org.apache.catalina.startup.Catalina stopServer
>
> SEVERE: Could not contact [localhost:8005] (base port [8005] and offset
> [0]). Tomcat may not be running.
>
> Nov 26, 2019 8:39:11 AM org.apache.catalina.startup.Catalina stopServer
>
> SEVERE: Error stopping Catalina
>
> java.net.ConnectException: Connection refused (Connection refused)
>
> at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
>
> at
> java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
>
> at
> java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
>
> at
> java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
>
> at java.base/java.net.Socket.connect(Socket.java:591)
>
> at java.base/java.net.Socket.connect(Socket.java:540)
>
> at java.base/java.net.Socket.<init>(Socket.java:436)
>
> at java.base/java.net.Socket.<init>(Socket.java:213)
>
> at
> org.apache.catalina.startup.Catalina.stopServer(Catalina.java:513)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:390)
>
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:480)
>
>
>
>
>
> OM5 log
>
>
>
> 26-Nov-2019 08:25:49.733 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
> [172.17.0.1]:5701 is SHUTDOWN
>
> 26-Nov-2019 08:25:49.790 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
> web application [Openmeetings] registered the JDBC driver
> [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
> application was stopped. To prevent a memory leak, the JDBC Driver has been
> forcibly unregistered.
>
> 26-Nov-2019 08:25:49.791 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
> 26-Nov-2019 08:25:49.792 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
> 26-Nov-2019 08:25:49.793 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10]
> but has failed to stop it. This is very likely to create a memory leak.
> Stack trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
> 26-Nov-2019 08:25:49.793 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [mysql-cj-abandoned-connection-cleanup] but has failed to stop it. This is
> very likely to create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
> java.base@11/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
>
>
> com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:85)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.794 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [pool-5-thread-1] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
>
> java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
>
>
> java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
>
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.795 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [pool-1-thread-1] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
>
> java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
>
>
> java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
>
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.795 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.796 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [pool-21-thread-1] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
>
> java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
>
>
> java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
>
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
>
>
> java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
>
>
> java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.796 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-2] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.797 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-3] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
>
>
> …..REPEATS SEVERAL TIMES
>
>
>
> 26-Nov-2019 08:25:49.814 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-30] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.815 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@560478b8]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.816 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a
> value of type [org.apache.wicket.ThreadContext] (value
> [org.apache.wicket.ThreadContext@12b21dea]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.816 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a
> value of type [org.apache.wicket.ThreadContext] (value
> [org.apache.wicket.ThreadContext@e41cc5d]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak…
>
>
>
> …REPEATS SEVERAL TIMES
>
>
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a
> value of type [org.apache.wicket.ThreadContext] (value
> [org.apache.wicket.ThreadContext@4e2fb8d6]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@1c5e617f]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@5147f2c6]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@ca5a0d3]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.823 INFO [main]
> org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler
> ["https-openssl-apr-443"]
>
> 26-Nov-2019 08:25:49.874 INFO [main]
> org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler
> ["ajp-nio-8009"]
>
> 26-Nov-2019 08:25:49.879 INFO [main]
> org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler
> ["https-openssl-apr-443"]
>
> 26-Nov-2019 08:25:49.881 INFO [main]
> org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler
> ["ajp-nio-8009"]
>
> 26-Nov-2019 08:25:49.913 INFO [mysql-cj-abandoned-connection-cleanup]
> org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading
> Illegal access: this web application instance has been stopped already.
> Could not load []. The following stack trace is thrown for debugging
> purposes as well as to attempt to terminate the thread which caused the
> illegal access.
>
> java.lang.IllegalStateException: Illegal access: this
> web application instance has been stopped already. Could not load []. The
> following stack trace is thrown for debugging purposes as well as to
> attempt to terminate the thread which caused the illegal access.
>
> at
> org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1385)
>
> at
> org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1038)
>
> at
> com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkThreadContextClassLoader(AbandonedConnectionCleanupThread.java:117)
>
> at
> com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:84)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at
> java.base/java.lang.Thread.run(Thread.java:834)
>
>
>
>
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> www.Coscend.com <http://www.coscend.com/>
>
> ------------------------------------------------------------------
>
> *Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education, Telepresence Services, on the fly…*
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
>
>
>
>
>
>
>
>
>
> *From:* Maxim Solodovnik [mailto:solomax666@gmail.com]
> *Sent:* Monday, November 25, 2019 10:21 AM
> *To:* Openmeetings user-list <us...@openmeetings.apache.org>;
> OM.Insights@coscend.com
> *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Hello Hemant,
>
>
>
> dev@ mailing list is for discussing OM/Om community development (i.e.
> discussing patch, the way to solve something, etc.)
>
> NOT for contacting OM devs to report an issue
>
>
>
> I don't have CentOS8 so I need detailed description of
>
> what do you mean by "Environment is CentOS 8 (using docker) and MariaDB
> 10.3.x"? (OM is java application, it doesn't require docker to be able to
> run)
>
>
>
> So could you run "vanilla" OM5 (without any other services, proxies, 3rd
> parties etc.) and check it it works for you as expected? i.e. both
> startup.sh and shutdown.sh run as correct user do it's job?
>
>
>
> If yes - add changes one by one and continue checking
>
> If not - compare with "vanilla" Tomcat9 - what is different :)
>
> baby steps :)
>
>
>
> On Fri, 22 Nov 2019 at 13:49, Coscend@OM <OM...@coscend.com> wrote:
>
> Further, if we do not run OM5, Tomcat shuts down properly. Environment is
> CentOS 8 (using docker) and MariaDB 10.3.x
>
>
>
> *From:* Coscend@OM [mailto:OM.Insights@Coscend.com]
> *Sent:* Friday, November 22, 2019 12:12 PM
> *To:* user@openmeetings.apache.org
> *Subject:* RE: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Dear Maxim,
>
>
>
> I am sending this to users, but we think one possibility (not
> confirmation) is this error is related to OM5 code [1]. There is a method
> in OM5 that is triggering memory leak while shutting down Tomcat. The most
> common reason is something hanging or delaying in Web application
> initializations [2]. This prevents Tomcat to properly shutdown [1, 2].
>
>
>
> Or, is due to something else? (We are using OM5 binary from OM Web site.
> There are no customization or configuration changes.)
>
>
>
> Here is the relevant log.
>
>
>
> The key line in the log is: “The web application [Openmeetings] appears
> to have started a thread named [nioEventLoopGroup-2-1] but has failed to
> stop it. This is very likely to create a memory leak.”
>
> ------------------------
>
> 21-Nov-2019 23:24:30.024 WARNING [main]
>
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
>
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
>
>
> 21-Nov-2019 23:24:30.025 WARNING [main]
>
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:java.base@11/java.lang.Object.wait(Native Method)
>
>
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
>
>
>
>
> [1]
> https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005
>
>
>
> [2]
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1
>
>
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> www.Coscend.com <http://www.coscend.com/>
>
> ------------------------------------------------------------------
>
> *Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education, Telepresence Services, on the fly…*
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
>
>
>
>
>
>
>
> *From:* Maxim Solodovnik [mailto:solomax666@gmail.com
> <so...@gmail.com>]
> *Sent:* Friday, November 22, 2019 11:36 AM
> *To:* dev <de...@openmeetings.apache.org>; OM.Insights@coscend.com
> *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Please send these messages to users@
>
> It is not about OM/OM_community development
>
>
>
> shutdown works for me as expected
>
> please provide additional details (if any)
>
>
>
> On Fri, 22 Nov 2019 at 13:04, Coscend@OM <OM...@coscend.com> wrote:
>
> Dear OM Community,
>
>
>
> While shutting down OM5-Tomcat via shutdown.sh, Tomcat is not shutting
> down.
> These links state some method in Webapp could be preventing the shutdown
> process of Tomcat:
>
>
> https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shu
> tdown-port-8005
>
>
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listen
> ing?noredirect=1
> <
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-liste
> ning?noredirect=1&lq=1> &lq=1
>
>
>
> Could anyone provide any guidance what we need to modify in OM5 code or any
> other place to properly shutdown Tomcat?
>
>
>
> Here is the log snippet. There are several more such errors while shutting
> down.
>
>
>
> ------------------------------
>
>
>
> 21-Nov-2019 23:24:29.843 INFO [main]
> org.apache.catalina.core.StandardServer.await A valid shutdown command was
> received via the shutdown port. Stopping the Server instance.
>
> 21-Nov-2019 23:24:29.844 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["http-nio-5080"]
>
> 21-Nov-2019 23:24:29.850 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["ajp-nio-8009"]
>
> 21-Nov-2019 23:24:29.856 INFO [main]
> org.apache.catalina.core.StandardService.stopInternal Stopping service
> [Catalina]
>
> [34mINFO [0;39m 11-21 23:24:29.884 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket extensions initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket core library initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy:
> org.wicketstuff.dashboard.DashboardContextInitializer@72690a1
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
> (theme-default)
>
> 21-Nov-2019 23:24:29.886 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
> [172.17.0.1]:5701 is SHUTTING_DOWN
>
> 21-Nov-2019 23:24:29.891 INFO [hz.server-1.cached.thread-2]
> com.hazelcast.internal.partition.impl.MigrationManager.null
> [172.17.0.1]:5701 [dev] [3.12] Shutdown request of Member [172.17.0.1]:5701
> - 799babea-5e78-4cf9-a099-f4b1fb09b9c9 this is handled
>
> 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Shutting down connection manager...
>
> 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Shutting down node engine...
>
> 21-Nov-2019 23:24:29.979 INFO [main]
> com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12]
> Destroying node NodeExtension.
>
> 21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Hazelcast Shutdown is completed in 90 ms.
>
> 21-Nov-2019 23:24:29.980 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
> [172.17.0.1]:5701 is SHUTDOWN
>
> 21-Nov-2019 23:24:30.023 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
> web
> application [Openmeetings] registered the JDBC driver
> [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
> application was stopped. To prevent a memory leak, the JDBC Driver has been
> forcibly unregistered.
>
> 21-Nov-2019 23:24:30.024 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThr
> eadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventEx
> ecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable
> .java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 21-Nov-2019 23:24:30.025 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555
> )
>
>
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> <http://www.coscend.com/> www.Coscend.com
>
> ------------------------------------------------------------------
>
> Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education,
> Telepresence Services, on the fly.
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
>
>
>
>
> --
>
> WBR
> Maxim aka solomax
>
>
>
>
> --
>
> WBR
> Maxim aka solomax
>
>
>
>
> --
>
> WBR
> Maxim aka solomax
>
--
WBR
Maxim aka solomax
RE: OM5: Tomcat not shutting down due to OM5 method
Posted by "Coscend@OM" <OM...@Coscend.com>.
Dear Maxim,
è The error "Could not contact [localhost:8005] " most probably means: you have disabled AJP. This is why shutdown.sh can do nothing
Our server.xml setting:
<Connector port="5443" protocol="org.apache.coyote.http11.Http11AprProtocol"
maxThreads="150" SSLEnabled="true" >
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
<SSLHostConfig>
<Certificate certificateKeyFile="/etc/….key"
certificateFile="/etc/...cer"
type="EC" />
</SSLHostConfig>
</Connector>
…
<Connector port="8009" protocol="AJP/1.3" redirectPort="<5443>" />
Question: There is a line in the log. After this line, there are several errors.
07-Dec-2019 16:55:45.082 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [Openmeetings] registered the JDBC driver [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
----------
Here is the OM log:
07-Dec-2019 16:55:44.938 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
07-Dec-2019 16:55:44.938 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-openssl-apr-5443"]
07-Dec-2019 16:55:44.945 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-8009"]
07-Dec-2019 16:55:44.950 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
[34mINFO [0;39m 12-07 16:55:44.983 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: org.wicketstuff.dashboard.DashboardContextInitializer@10c541dc
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer (theme-default)
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket core library initializer
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
[34mINFO [0;39m 12-07 16:55:44.984 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket extensions initializer
07-Dec-2019 16:55:44.984 INFO [main] com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12.2] [172.17.0.1]:5701 is SHUTTING_DOWN
07-Dec-2019 16:55:44.990 INFO [hz.server-1.cached.thread-4] com.hazelcast.internal.partition.impl.MigrationManager.null [172.17.0.1]:5701 [dev] [3.12.2] Shutdown request of Member [172.17.0.1]:5701 - 7d3e2187-936e-49b1-a0b7-cdb0f2a604d6 this is handled
07-Dec-2019 16:55:44.996 INFO [main] com.hazelcast.instance.Node.null [172.17.0.1]:5701 [dev] [3.12.2] Shutting down connection manager...
07-Dec-2019 16:55:44.997 INFO [main] com.hazelcast.instance.Node.null [172.17.0.1]:5701 [dev] [3.12.2] Shutting down node engine...
07-Dec-2019 16:55:45.011 INFO [main] com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12.2] Destroying node NodeExtension.
07-Dec-2019 16:55:45.011 INFO [main] com.hazelcast.instance.Node.null [172.17.0.1]:5701 [dev] [3.12.2] Hazelcast Shutdown is completed in 22 ms.
07-Dec-2019 16:55:45.011 INFO [main] com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12.2] [172.17.0.1]:5701 is SHUTDOWN
07-Dec-2019 16:55:45.082 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [Openmeetings] registered the JDBC driver [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
07-Dec-2019 16:55:45.083 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:720)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:510)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834)
07-Dec-2019 16:55:45.083 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly…
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
From: Maxim Solodovnik [mailto:solomax666@gmail.com]
Sent: Tuesday, November 26, 2019 9:00 PM
To: Openmeetings user-list <us...@openmeetings.apache.org>; OM.Insights@coscend.com
Subject: Re: OM5: Tomcat not shutting down due to OM5 method
Hello Hemant,
Vanilla OM5 doen't have MySQL driver
It also not uses port 443
I would suggest to test OM with no modifications (before ANY customizations)
The error "Could not contact [localhost:8005] " most probably means: you have disabled AJP
This is why shutdown.sh can do nothing
you can still use `kill -9` in this case
On Tue, 26 Nov 2019 at 21:59, Coscend@OM <OM.Insights@coscend.com <ma...@coscend.com> > wrote:
Dear Maxim,
Vanilla Tomcat9: startup and shutdown fine.
Vanilla OM5
User: when the binary is unarchived, it has a local usergroup as the default owner. Is this correct?
(binary from OpenMeetings Web site): no configuration changes except SSL certificate input.
Startup fine
Shutdown:
netstat gives no port binding on 443 and no pid
echo $CATALINA_PID gives blank
BUT
ps aux gives tomcat pid running
Below are CLI log and OM5 / Catalina log.
If we run shutdown command again, CLI gives “Tomcat may not be running”:
-------------------------------------------------------
CLI log
org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact [localhost:8005] (base port [8005] and offset [0]). Tomcat may not be running.
Nov 26, 2019 8:39:11 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Error stopping Catalina
java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at java.base/java.net.Socket.connect(Socket.java:540)
at java.base/java.net.Socket.<init>(Socket.java:436)
at java.base/java.net.Socket.<init>(Socket.java:213)
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:513)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:390)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:480)
OM5 log
26-Nov-2019 08:25:49.733 INFO [main] com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12] [172.17.0.1]:5701 is SHUTDOWN
26-Nov-2019 08:25:49.790 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [Openmeetings] registered the JDBC driver [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
26-Nov-2019 08:25:49.791 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
26-Nov-2019 08:25:49.792 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
26-Nov-2019 08:25:49.793 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
26-Nov-2019 08:25:49.793 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [mysql-cj-abandoned-connection-cleanup] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
java.base@11/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155) <mailto:java.base@11/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)>
com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:85)
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)>
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.794 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/jdk.internal.misc.Unsafe.park(Native <mailto:java.base@11/jdk.internal.misc.Unsafe.park(Native> Method)
java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) <mailto:java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)>
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) <mailto:java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)>
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.795 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/jdk.internal.misc.Unsafe.park(Native <mailto:java.base@11/jdk.internal.misc.Unsafe.park(Native> Method)
java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) <mailto:java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)>
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) <mailto:java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)>
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.795 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.796 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [pool-21-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/jdk.internal.misc.Unsafe.park(Native <mailto:java.base@11/jdk.internal.misc.Unsafe.park(Native> Method)
java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) <mailto:java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)>
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) <mailto:java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)>
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) <mailto:java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)>
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)>
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) <mailto:java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.796 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.797 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
…..REPEATS SEVERAL TIMES
26-Nov-2019 08:25:49.814 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-30] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
26-Nov-2019 08:25:49.815 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@560478b8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.816 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a value of type [org.apache.wicket.ThreadContext] (value [org.apache.wicket.ThreadContext@12b21dea]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.816 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a value of type [org.apache.wicket.ThreadContext] (value [org.apache.wicket.ThreadContext@e41cc5d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak…
…REPEATS SEVERAL TIMES
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a value of type [org.apache.wicket.ThreadContext] (value [org.apache.wicket.ThreadContext@4e2fb8d6]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1c5e617f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@5147f2c6]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@ca5a0d3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.823 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-openssl-apr-443"]
26-Nov-2019 08:25:49.874 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-8009"]
26-Nov-2019 08:25:49.879 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-openssl-apr-443"]
26-Nov-2019 08:25:49.881 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-8009"]
26-Nov-2019 08:25:49.913 INFO [mysql-cj-abandoned-connection-cleanup] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load []. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load []. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1385)
at org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1038)
at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkThreadContextClassLoader(AbandonedConnectionCleanupThread.java:117)
at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:84)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Thank you.
Sincerely,
Hemant K. Sabat
www.Coscend.com <http://www.coscend.com/>
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly…
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
From: Maxim Solodovnik [mailto:solomax666@gmail.com <ma...@gmail.com> ]
Sent: Monday, November 25, 2019 10:21 AM
To: Openmeetings user-list <user@openmeetings.apache.org <ma...@openmeetings.apache.org> >; OM.Insights@coscend.com <ma...@coscend.com>
Subject: Re: OM5: Tomcat not shutting down due to OM5 method
Hello Hemant,
dev@ mailing list is for discussing OM/Om community development (i.e. discussing patch, the way to solve something, etc.)
NOT for contacting OM devs to report an issue
I don't have CentOS8 so I need detailed description of
what do you mean by "Environment is CentOS 8 (using docker) and MariaDB 10.3.x"? (OM is java application, it doesn't require docker to be able to run)
So could you run "vanilla" OM5 (without any other services, proxies, 3rd parties etc.) and check it it works for you as expected? i.e. both startup.sh and shutdown.sh run as correct user do it's job?
If yes - add changes one by one and continue checking
If not - compare with "vanilla" Tomcat9 - what is different :)
baby steps :)
On Fri, 22 Nov 2019 at 13:49, Coscend@OM <OM.Insights@coscend.com <ma...@coscend.com> > wrote:
Further, if we do not run OM5, Tomcat shuts down properly. Environment is CentOS 8 (using docker) and MariaDB 10.3.x
From: Coscend@OM [mailto:OM.Insights@Coscend.com <ma...@Coscend.com> ]
Sent: Friday, November 22, 2019 12:12 PM
To: user@openmeetings.apache.org <ma...@openmeetings.apache.org>
Subject: RE: OM5: Tomcat not shutting down due to OM5 method
Dear Maxim,
I am sending this to users, but we think one possibility (not confirmation) is this error is related to OM5 code [1]. There is a method in OM5 that is triggering memory leak while shutting down Tomcat. The most common reason is something hanging or delaying in Web application initializations [2]. This prevents Tomcat to properly shutdown [1, 2].
Or, is due to something else? (We are using OM5 binary from OM Web site. There are no customization or configuration changes.)
Here is the relevant log.
The key line in the log is: “The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak.”
------------------------
21-Nov-2019 23:24:30.024 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
21-Nov-2019 23:24:30.025 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
[1] https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005
[2] https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly…
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
From: Maxim Solodovnik [mailto:solomax666@gmail.com]
Sent: Friday, November 22, 2019 11:36 AM
To: dev <dev@openmeetings.apache.org <ma...@openmeetings.apache.org> >; OM.Insights@coscend.com <ma...@coscend.com>
Subject: Re: OM5: Tomcat not shutting down due to OM5 method
Please send these messages to users@
It is not about OM/OM_community development
shutdown works for me as expected
please provide additional details (if any)
On Fri, 22 Nov 2019 at 13:04, Coscend@OM <OM.Insights@coscend.com <ma...@coscend.com> > wrote:
Dear OM Community,
While shutting down OM5-Tomcat via shutdown.sh, Tomcat is not shutting down.
These links state some method in Webapp could be preventing the shutdown
process of Tomcat:
https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shu <https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005>
tdown-port-8005
https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listen <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1>
ing?noredirect=1
<https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-liste <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1&lq=1>
ning?noredirect=1&lq=1> &lq=1
Could anyone provide any guidance what we need to modify in OM5 code or any
other place to properly shutdown Tomcat?
Here is the log snippet. There are several more such errors while shutting
down.
------------------------------
21-Nov-2019 23:24:29.843 INFO [main]
org.apache.catalina.core.StandardServer.await A valid shutdown command was
received via the shutdown port. Stopping the Server instance.
21-Nov-2019 23:24:29.844 INFO [main]
org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
["http-nio-5080"]
21-Nov-2019 23:24:29.850 INFO [main]
org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
["ajp-nio-8009"]
21-Nov-2019 23:24:29.856 INFO [main]
org.apache.catalina.core.StandardService.stopInternal Stopping service
[Catalina]
[34mINFO [0;39m 11-21 23:24:29.884 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket extensions initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket core library initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy:
org.wicketstuff.dashboard.DashboardContextInitializer@72690a1 <ma...@72690a1>
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
(theme-default)
21-Nov-2019 23:24:29.886 INFO [main]
com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
[172.17.0.1]:5701 is SHUTTING_DOWN
21-Nov-2019 23:24:29.891 INFO [hz.server-1.cached.thread-2]
com.hazelcast.internal.partition.impl.MigrationManager.null
[172.17.0.1]:5701 [dev] [3.12] Shutdown request of Member [172.17.0.1]:5701
- 799babea-5e78-4cf9-a099-f4b1fb09b9c9 this is handled
21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Shutting down connection manager...
21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Shutting down node engine...
21-Nov-2019 23:24:29.979 INFO [main]
com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12]
Destroying node NodeExtension.
21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Hazelcast Shutdown is completed in 90 ms.
21-Nov-2019 23:24:29.980 INFO [main]
com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
[172.17.0.1]:5701 is SHUTDOWN
21-Nov-2019 23:24:30.023 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web
application [Openmeetings] registered the JDBC driver
[com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
application was stopped. To prevent a memory leak, the JDBC Driver has been
forcibly unregistered.
21-Nov-2019 23:24:30.024 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [Openmeetings] appears to have started a thread named
[nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThr
eadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventEx
ecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable
.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
21-Nov-2019 23:24:30.025 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [Openmeetings] appears to have started a thread named
[org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
has failed to stop it. This is very likely to create a memory leak. Stack
trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555
)
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com <http://www.Coscend.com>
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education,
Telepresence Services, on the fly.
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
Messages from Coscend Communications Solutions' posted at:
<http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
--
WBR
Maxim aka solomax
--
WBR
Maxim aka solomax
--
WBR
Maxim aka solomax
Re: OM5: Tomcat not shutting down due to OM5 method
Posted by Maxim Solodovnik <so...@gmail.com>.
Hello Hemant,
Vanilla OM5 doen't have MySQL driver
It also not uses port 443
I would suggest to test OM with no modifications (before ANY customizations)
The error "Could not contact [localhost:8005] " most probably means: you
have disabled AJP
This is why shutdown.sh can do nothing
you can still use `kill -9` in this case
On Tue, 26 Nov 2019 at 21:59, Coscend@OM <OM...@coscend.com> wrote:
> Dear Maxim,
>
>
>
> Vanilla Tomcat9: startup and shutdown fine.
>
>
>
> Vanilla OM5
>
> User: when the binary is unarchived, it has a local usergroup as the
> default owner. Is this correct?
>
>
>
> (binary from OpenMeetings Web site): no configuration changes except SSL
> certificate input.
>
>
>
>
>
> Startup fine
>
> Shutdown:
>
> netstat gives no port binding on 443 and no pid
>
> echo $CATALINA_PID gives blank
>
> BUT
>
> ps aux gives tomcat pid running
>
>
>
> Below are CLI log and OM5 / Catalina log.
>
>
>
>
>
> If we run shutdown command again, CLI gives “Tomcat may not be running”:
>
>
>
> -------------------------------------------------------
>
> CLI log
>
>
>
> org.apache.catalina.startup.Catalina stopServer
>
> SEVERE: Could not contact [localhost:8005] (base port [8005] and offset
> [0]). Tomcat may not be running.
>
> Nov 26, 2019 8:39:11 AM org.apache.catalina.startup.Catalina stopServer
>
> SEVERE: Error stopping Catalina
>
> java.net.ConnectException: Connection refused (Connection refused)
>
> at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
>
> at
> java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
>
> at
> java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
>
> at
> java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
>
> at java.base/java.net.Socket.connect(Socket.java:591)
>
> at java.base/java.net.Socket.connect(Socket.java:540)
>
> at java.base/java.net.Socket.<init>(Socket.java:436)
>
> at java.base/java.net.Socket.<init>(Socket.java:213)
>
> at
> org.apache.catalina.startup.Catalina.stopServer(Catalina.java:513)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:390)
>
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:480)
>
>
>
>
>
> OM5 log
>
>
>
> 26-Nov-2019 08:25:49.733 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
> [172.17.0.1]:5701 is SHUTDOWN
>
> 26-Nov-2019 08:25:49.790 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
> web application [Openmeetings] registered the JDBC driver
> [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
> application was stopped. To prevent a memory leak, the JDBC Driver has been
> forcibly unregistered.
>
> 26-Nov-2019 08:25:49.791 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
> 26-Nov-2019 08:25:49.792 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
> 26-Nov-2019 08:25:49.793 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10]
> but has failed to stop it. This is very likely to create a memory leak.
> Stack trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
> 26-Nov-2019 08:25:49.793 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [mysql-cj-abandoned-connection-cleanup] but has failed to stop it. This is
> very likely to create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
> java.base@11/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
>
>
> com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:85)
>
> java.base@11
> /java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> java.base@11
> /java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.794 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [pool-5-thread-1] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
>
> java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
>
> java.base@11
> /java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
> java.base@11
> /java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
>
> java.base@11
> /java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
>
> java.base@11
> /java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
>
> java.base@11
> /java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
>
> java.base@11
> /java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
>
> java.base@11
> /java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.795 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [pool-1-thread-1] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
>
> java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
>
> java.base@11
> /java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
> java.base@11
> /java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
>
> java.base@11
> /java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
>
> java.base@11
> /java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
>
> java.base@11
> /java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
>
> java.base@11
> /java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
>
> java.base@11
> /java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.795 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.796 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [pool-21-thread-1] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
>
> java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
>
> java.base@11
> /java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
>
> java.base@11
> /java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
>
> java.base@11
> /java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
>
> java.base@11
> /java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
>
> java.base@11
> /java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
>
> java.base@11
> /java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
>
> java.base@11
> /java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.796 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-2] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.797 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-3] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
>
>
> …..REPEATS SEVERAL TIMES
>
>
>
> 26-Nov-2019 08:25:49.814 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-6-30] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 26-Nov-2019 08:25:49.815 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@560478b8]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.816 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a
> value of type [org.apache.wicket.ThreadContext] (value
> [org.apache.wicket.ThreadContext@12b21dea]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.816 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a
> value of type [org.apache.wicket.ThreadContext] (value
> [org.apache.wicket.ThreadContext@e41cc5d]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak…
>
>
>
> …REPEATS SEVERAL TIMES
>
>
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a
> value of type [org.apache.wicket.ThreadContext] (value
> [org.apache.wicket.ThreadContext@4e2fb8d6]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@1c5e617f]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@5147f2c6]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.818 SEVERE [main]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [Openmeetings] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a
> value of type [io.netty.util.internal.InternalThreadLocalMap] (value
> [io.netty.util.internal.InternalThreadLocalMap@ca5a0d3]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
>
> 26-Nov-2019 08:25:49.823 INFO [main]
> org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler
> ["https-openssl-apr-443"]
>
> 26-Nov-2019 08:25:49.874 INFO [main]
> org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler
> ["ajp-nio-8009"]
>
> 26-Nov-2019 08:25:49.879 INFO [main]
> org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler
> ["https-openssl-apr-443"]
>
> 26-Nov-2019 08:25:49.881 INFO [main]
> org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler
> ["ajp-nio-8009"]
>
> 26-Nov-2019 08:25:49.913 INFO [mysql-cj-abandoned-connection-cleanup]
> org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading
> Illegal access: this web application instance has been stopped already.
> Could not load []. The following stack trace is thrown for debugging
> purposes as well as to attempt to terminate the thread which caused the
> illegal access.
>
> java.lang.IllegalStateException: Illegal access: this
> web application instance has been stopped already. Could not load []. The
> following stack trace is thrown for debugging purposes as well as to
> attempt to terminate the thread which caused the illegal access.
>
> at
> org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1385)
>
> at
> org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1038)
>
> at
> com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkThreadContextClassLoader(AbandonedConnectionCleanupThread.java:117)
>
> at
> com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:84)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at
> java.base/java.lang.Thread.run(Thread.java:834)
>
>
>
>
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> www.Coscend.com <http://www.coscend.com/>
>
> ------------------------------------------------------------------
>
> *Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education, Telepresence Services, on the fly…*
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
>
>
>
>
>
>
>
>
>
> *From:* Maxim Solodovnik [mailto:solomax666@gmail.com]
> *Sent:* Monday, November 25, 2019 10:21 AM
> *To:* Openmeetings user-list <us...@openmeetings.apache.org>;
> OM.Insights@coscend.com
> *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Hello Hemant,
>
>
>
> dev@ mailing list is for discussing OM/Om community development (i.e.
> discussing patch, the way to solve something, etc.)
>
> NOT for contacting OM devs to report an issue
>
>
>
> I don't have CentOS8 so I need detailed description of
>
> what do you mean by "Environment is CentOS 8 (using docker) and MariaDB
> 10.3.x"? (OM is java application, it doesn't require docker to be able to
> run)
>
>
>
> So could you run "vanilla" OM5 (without any other services, proxies, 3rd
> parties etc.) and check it it works for you as expected? i.e. both
> startup.sh and shutdown.sh run as correct user do it's job?
>
>
>
> If yes - add changes one by one and continue checking
>
> If not - compare with "vanilla" Tomcat9 - what is different :)
>
> baby steps :)
>
>
>
> On Fri, 22 Nov 2019 at 13:49, Coscend@OM <OM...@coscend.com> wrote:
>
> Further, if we do not run OM5, Tomcat shuts down properly. Environment is
> CentOS 8 (using docker) and MariaDB 10.3.x
>
>
>
> *From:* Coscend@OM [mailto:OM.Insights@Coscend.com]
> *Sent:* Friday, November 22, 2019 12:12 PM
> *To:* user@openmeetings.apache.org
> *Subject:* RE: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Dear Maxim,
>
>
>
> I am sending this to users, but we think one possibility (not
> confirmation) is this error is related to OM5 code [1]. There is a method
> in OM5 that is triggering memory leak while shutting down Tomcat. The most
> common reason is something hanging or delaying in Web application
> initializations [2]. This prevents Tomcat to properly shutdown [1, 2].
>
>
>
> Or, is due to something else? (We are using OM5 binary from OM Web site.
> There are no customization or configuration changes.)
>
>
>
> Here is the relevant log.
>
>
>
> The key line in the log is: “The web application [Openmeetings] appears
> to have started a thread named [nioEventLoopGroup-2-1] but has failed to
> stop it. This is very likely to create a memory leak.”
>
> ------------------------
>
> 21-Nov-2019 23:24:30.024 WARNING [main]
>
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
>
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
>
>
> 21-Nov-2019 23:24:30.025 WARNING [main]
>
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:java.base@11/java.lang.Object.wait(Native Method)
>
>
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
>
>
>
>
> [1]
> https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005
>
>
>
> [2]
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1
>
>
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> www.Coscend.com <http://www.coscend.com/>
>
> ------------------------------------------------------------------
>
> *Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education, Telepresence Services, on the fly…*
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
>
>
>
>
>
>
>
> *From:* Maxim Solodovnik [mailto:solomax666@gmail.com
> <so...@gmail.com>]
> *Sent:* Friday, November 22, 2019 11:36 AM
> *To:* dev <de...@openmeetings.apache.org>; OM.Insights@coscend.com
> *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Please send these messages to users@
>
> It is not about OM/OM_community development
>
>
>
> shutdown works for me as expected
>
> please provide additional details (if any)
>
>
>
> On Fri, 22 Nov 2019 at 13:04, Coscend@OM <OM...@coscend.com> wrote:
>
> Dear OM Community,
>
>
>
> While shutting down OM5-Tomcat via shutdown.sh, Tomcat is not shutting
> down.
> These links state some method in Webapp could be preventing the shutdown
> process of Tomcat:
>
>
> https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shu
> tdown-port-8005
>
>
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listen
> ing?noredirect=1
> <
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-liste
> ning?noredirect=1&lq=1> &lq=1
>
>
>
> Could anyone provide any guidance what we need to modify in OM5 code or any
> other place to properly shutdown Tomcat?
>
>
>
> Here is the log snippet. There are several more such errors while shutting
> down.
>
>
>
> ------------------------------
>
>
>
> 21-Nov-2019 23:24:29.843 INFO [main]
> org.apache.catalina.core.StandardServer.await A valid shutdown command was
> received via the shutdown port. Stopping the Server instance.
>
> 21-Nov-2019 23:24:29.844 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["http-nio-5080"]
>
> 21-Nov-2019 23:24:29.850 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["ajp-nio-8009"]
>
> 21-Nov-2019 23:24:29.856 INFO [main]
> org.apache.catalina.core.StandardService.stopInternal Stopping service
> [Catalina]
>
> [34mINFO [0;39m 11-21 23:24:29.884 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket extensions initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket core library initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy:
> org.wicketstuff.dashboard.DashboardContextInitializer@72690a1
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
> (theme-default)
>
> 21-Nov-2019 23:24:29.886 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
> [172.17.0.1]:5701 is SHUTTING_DOWN
>
> 21-Nov-2019 23:24:29.891 INFO [hz.server-1.cached.thread-2]
> com.hazelcast.internal.partition.impl.MigrationManager.null
> [172.17.0.1]:5701 [dev] [3.12] Shutdown request of Member [172.17.0.1]:5701
> - 799babea-5e78-4cf9-a099-f4b1fb09b9c9 this is handled
>
> 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Shutting down connection manager...
>
> 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Shutting down node engine...
>
> 21-Nov-2019 23:24:29.979 INFO [main]
> com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12]
> Destroying node NodeExtension.
>
> 21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Hazelcast Shutdown is completed in 90 ms.
>
> 21-Nov-2019 23:24:29.980 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
> [172.17.0.1]:5701 is SHUTDOWN
>
> 21-Nov-2019 23:24:30.023 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
> web
> application [Openmeetings] registered the JDBC driver
> [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
> application was stopped. To prevent a memory leak, the JDBC Driver has been
> forcibly unregistered.
>
> 21-Nov-2019 23:24:30.024 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThr
> eadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventEx
> ecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable
> .java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 21-Nov-2019 23:24:30.025 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555
> )
>
>
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> <http://www.coscend.com/> www.Coscend.com
>
> ------------------------------------------------------------------
>
> Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education,
> Telepresence Services, on the fly.
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
>
>
>
>
>
> --
>
> WBR
> Maxim aka solomax
>
>
>
>
> --
>
> WBR
> Maxim aka solomax
>
--
WBR
Maxim aka solomax
RE: OM5: Tomcat not shutting down due to OM5 method
Posted by "Coscend@OM" <OM...@Coscend.com>.
Dear Maxim,
Vanilla Tomcat9: startup and shutdown fine.
Vanilla OM5
User: when the binary is unarchived, it has a local usergroup as the default owner. Is this correct?
(binary from OpenMeetings Web site): no configuration changes except SSL certificate input.
Startup fine
Shutdown:
netstat gives no port binding on 443 and no pid
echo $CATALINA_PID gives blank
BUT
ps aux gives tomcat pid running
Below are CLI log and OM5 / Catalina log.
If we run shutdown command again, CLI gives “Tomcat may not be running”:
-------------------------------------------------------
CLI log
org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact [localhost:8005] (base port [8005] and offset [0]). Tomcat may not be running.
Nov 26, 2019 8:39:11 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Error stopping Catalina
java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at java.base/java.net.Socket.connect(Socket.java:540)
at java.base/java.net.Socket.<init>(Socket.java:436)
at java.base/java.net.Socket.<init>(Socket.java:213)
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:513)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:390)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:480)
OM5 log
26-Nov-2019 08:25:49.733 INFO [main] com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12] [172.17.0.1]:5701 is SHUTDOWN
26-Nov-2019 08:25:49.790 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [Openmeetings] registered the JDBC driver [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
26-Nov-2019 08:25:49.791 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
26-Nov-2019 08:25:49.792 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
26-Nov-2019 08:25:49.793 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
26-Nov-2019 08:25:49.793 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [mysql-cj-abandoned-connection-cleanup] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Object.wait(Native Method)
java.base@11/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:85)
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@11/java.lang.Thread.run(Thread.java:834)
26-Nov-2019 08:25:49.794 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@11/java.lang.Thread.run(Thread.java:834)
26-Nov-2019 08:25:49.795 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@11/java.lang.Thread.run(Thread.java:834)
26-Nov-2019 08:25:49.795 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834)
26-Nov-2019 08:25:49.796 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [pool-21-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@11/java.lang.Thread.run(Thread.java:834)
26-Nov-2019 08:25:49.796 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834)
26-Nov-2019 08:25:49.797 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834)
…..REPEATS SEVERAL TIMES
26-Nov-2019 08:25:49.814 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-6-30] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834)
26-Nov-2019 08:25:49.815 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@560478b8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.816 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a value of type [org.apache.wicket.ThreadContext] (value [org.apache.wicket.ThreadContext@12b21dea]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.816 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a value of type [org.apache.wicket.ThreadContext] (value [org.apache.wicket.ThreadContext@e41cc5d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak…
…REPEATS SEVERAL TIMES
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@71da8156]) and a value of type [org.apache.wicket.ThreadContext] (value [org.apache.wicket.ThreadContext@4e2fb8d6]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1c5e617f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@5147f2c6]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.818 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Openmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@724dc44d]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@ca5a0d3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
26-Nov-2019 08:25:49.823 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-openssl-apr-443"]
26-Nov-2019 08:25:49.874 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-8009"]
26-Nov-2019 08:25:49.879 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-openssl-apr-443"]
26-Nov-2019 08:25:49.881 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-8009"]
26-Nov-2019 08:25:49.913 INFO [mysql-cj-abandoned-connection-cleanup] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load []. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load []. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1385)
at org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1038)
at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkThreadContextClassLoader(AbandonedConnectionCleanupThread.java:117)
at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:84)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Thank you.
Sincerely,
Hemant K. Sabat
www.Coscend.com <http://www.coscend.com/>
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly…
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
From: Maxim Solodovnik [mailto:solomax666@gmail.com]
Sent: Monday, November 25, 2019 10:21 AM
To: Openmeetings user-list <us...@openmeetings.apache.org>; OM.Insights@coscend.com
Subject: Re: OM5: Tomcat not shutting down due to OM5 method
Hello Hemant,
dev@ mailing list is for discussing OM/Om community development (i.e. discussing patch, the way to solve something, etc.)
NOT for contacting OM devs to report an issue
I don't have CentOS8 so I need detailed description of
what do you mean by "Environment is CentOS 8 (using docker) and MariaDB 10.3.x"? (OM is java application, it doesn't require docker to be able to run)
So could you run "vanilla" OM5 (without any other services, proxies, 3rd parties etc.) and check it it works for you as expected? i.e. both startup.sh and shutdown.sh run as correct user do it's job?
If yes - add changes one by one and continue checking
If not - compare with "vanilla" Tomcat9 - what is different :)
baby steps :)
On Fri, 22 Nov 2019 at 13:49, Coscend@OM <OM.Insights@coscend.com <ma...@coscend.com> > wrote:
Further, if we do not run OM5, Tomcat shuts down properly. Environment is CentOS 8 (using docker) and MariaDB 10.3.x
From: Coscend@OM [mailto:OM.Insights@Coscend.com <ma...@Coscend.com> ]
Sent: Friday, November 22, 2019 12:12 PM
To: user@openmeetings.apache.org <ma...@openmeetings.apache.org>
Subject: RE: OM5: Tomcat not shutting down due to OM5 method
Dear Maxim,
I am sending this to users, but we think one possibility (not confirmation) is this error is related to OM5 code [1]. There is a method in OM5 that is triggering memory leak while shutting down Tomcat. The most common reason is something hanging or delaying in Web application initializations [2]. This prevents Tomcat to properly shutdown [1, 2].
Or, is due to something else? (We are using OM5 binary from OM Web site. There are no customization or configuration changes.)
Here is the relevant log.
The key line in the log is: “The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak.”
------------------------
21-Nov-2019 23:24:30.024 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
21-Nov-2019 23:24:30.025 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
[1] https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005
[2] https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly…
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
From: Maxim Solodovnik [mailto:solomax666@gmail.com]
Sent: Friday, November 22, 2019 11:36 AM
To: dev <dev@openmeetings.apache.org <ma...@openmeetings.apache.org> >; OM.Insights@coscend.com <ma...@coscend.com>
Subject: Re: OM5: Tomcat not shutting down due to OM5 method
Please send these messages to users@
It is not about OM/OM_community development
shutdown works for me as expected
please provide additional details (if any)
On Fri, 22 Nov 2019 at 13:04, Coscend@OM <OM.Insights@coscend.com <ma...@coscend.com> > wrote:
Dear OM Community,
While shutting down OM5-Tomcat via shutdown.sh, Tomcat is not shutting down.
These links state some method in Webapp could be preventing the shutdown
process of Tomcat:
https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shu <https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005>
tdown-port-8005
https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listen <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1>
ing?noredirect=1
<https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-liste <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1&lq=1>
ning?noredirect=1&lq=1> &lq=1
Could anyone provide any guidance what we need to modify in OM5 code or any
other place to properly shutdown Tomcat?
Here is the log snippet. There are several more such errors while shutting
down.
------------------------------
21-Nov-2019 23:24:29.843 INFO [main]
org.apache.catalina.core.StandardServer.await A valid shutdown command was
received via the shutdown port. Stopping the Server instance.
21-Nov-2019 23:24:29.844 INFO [main]
org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
["http-nio-5080"]
21-Nov-2019 23:24:29.850 INFO [main]
org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
["ajp-nio-8009"]
21-Nov-2019 23:24:29.856 INFO [main]
org.apache.catalina.core.StandardService.stopInternal Stopping service
[Catalina]
[34mINFO [0;39m 11-21 23:24:29.884 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket extensions initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket core library initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy:
org.wicketstuff.dashboard.DashboardContextInitializer@72690a1 <ma...@72690a1>
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
(theme-default)
21-Nov-2019 23:24:29.886 INFO [main]
com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
[172.17.0.1]:5701 is SHUTTING_DOWN
21-Nov-2019 23:24:29.891 INFO [hz.server-1.cached.thread-2]
com.hazelcast.internal.partition.impl.MigrationManager.null
[172.17.0.1]:5701 [dev] [3.12] Shutdown request of Member [172.17.0.1]:5701
- 799babea-5e78-4cf9-a099-f4b1fb09b9c9 this is handled
21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Shutting down connection manager...
21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Shutting down node engine...
21-Nov-2019 23:24:29.979 INFO [main]
com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12]
Destroying node NodeExtension.
21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Hazelcast Shutdown is completed in 90 ms.
21-Nov-2019 23:24:29.980 INFO [main]
com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
[172.17.0.1]:5701 is SHUTDOWN
21-Nov-2019 23:24:30.023 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web
application [Openmeetings] registered the JDBC driver
[com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
application was stopped. To prevent a memory leak, the JDBC Driver has been
forcibly unregistered.
21-Nov-2019 23:24:30.024 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [Openmeetings] appears to have started a thread named
[nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThr
eadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventEx
ecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable
.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
21-Nov-2019 23:24:30.025 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [Openmeetings] appears to have started a thread named
[org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
has failed to stop it. This is very likely to create a memory leak. Stack
trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555
)
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com <http://www.Coscend.com>
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education,
Telepresence Services, on the fly.
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
Messages from Coscend Communications Solutions' posted at:
<http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
--
WBR
Maxim aka solomax
--
WBR
Maxim aka solomax
Re: OM5: Tomcat not shutting down due to OM5 method
Posted by Maxim Solodovnik <so...@gmail.com>.
Hello Hemant,
dev@ mailing list is for discussing OM/Om community development (i.e.
discussing patch, the way to solve something, etc.)
NOT for contacting OM devs to report an issue
I don't have CentOS8 so I need detailed description of
what do you mean by "Environment is CentOS 8 (using docker) and MariaDB
10.3.x"? (OM is java application, it doesn't require docker to be able to
run)
So could you run "vanilla" OM5 (without any other services, proxies, 3rd
parties etc.) and check it it works for you as expected? i.e. both
startup.sh and shutdown.sh run as correct user do it's job?
If yes - add changes one by one and continue checking
If not - compare with "vanilla" Tomcat9 - what is different :)
baby steps :)
On Fri, 22 Nov 2019 at 13:49, Coscend@OM <OM...@coscend.com> wrote:
> Further, if we do not run OM5, Tomcat shuts down properly. Environment is
> CentOS 8 (using docker) and MariaDB 10.3.x
>
>
>
> *From:* Coscend@OM [mailto:OM.Insights@Coscend.com]
> *Sent:* Friday, November 22, 2019 12:12 PM
> *To:* user@openmeetings.apache.org
> *Subject:* RE: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Dear Maxim,
>
>
>
> I am sending this to users, but we think one possibility (not
> confirmation) is this error is related to OM5 code [1]. There is a method
> in OM5 that is triggering memory leak while shutting down Tomcat. The most
> common reason is something hanging or delaying in Web application
> initializations [2]. This prevents Tomcat to properly shutdown [1, 2].
>
>
>
> Or, is due to something else? (We are using OM5 binary from OM Web site.
> There are no customization or configuration changes.)
>
>
>
> Here is the relevant log.
>
>
>
> The key line in the log is: “The web application [Openmeetings] appears
> to have started a thread named [nioEventLoopGroup-2-1] but has failed to
> stop it. This is very likely to create a memory leak.”
>
> ------------------------
>
> 21-Nov-2019 23:24:30.024 WARNING [main]
>
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
>
>
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
>
>
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
>
>
> 21-Nov-2019 23:24:30.025 WARNING [main]
>
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:java.base@11/java.lang.Object.wait(Native Method)
>
>
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
>
>
>
>
>
> [1]
> https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005
>
>
>
> [2]
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1
>
>
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> www.Coscend.com <http://www.coscend.com/>
>
> ------------------------------------------------------------------
>
> *Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education, Telepresence Services, on the fly…*
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
>
>
>
>
>
>
>
> *From:* Maxim Solodovnik [mailto:solomax666@gmail.com
> <so...@gmail.com>]
> *Sent:* Friday, November 22, 2019 11:36 AM
> *To:* dev <de...@openmeetings.apache.org>; OM.Insights@coscend.com
> *Subject:* Re: OM5: Tomcat not shutting down due to OM5 method
>
>
>
> Please send these messages to users@
>
> It is not about OM/OM_community development
>
>
>
> shutdown works for me as expected
>
> please provide additional details (if any)
>
>
>
> On Fri, 22 Nov 2019 at 13:04, Coscend@OM <OM...@coscend.com> wrote:
>
> Dear OM Community,
>
>
>
> While shutting down OM5-Tomcat via shutdown.sh, Tomcat is not shutting
> down.
> These links state some method in Webapp could be preventing the shutdown
> process of Tomcat:
>
>
> https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shu
> tdown-port-8005
>
>
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listen
> ing?noredirect=1
> <
> https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-liste
> ning?noredirect=1&lq=1> &lq=1
>
>
>
> Could anyone provide any guidance what we need to modify in OM5 code or any
> other place to properly shutdown Tomcat?
>
>
>
> Here is the log snippet. There are several more such errors while shutting
> down.
>
>
>
> ------------------------------
>
>
>
> 21-Nov-2019 23:24:29.843 INFO [main]
> org.apache.catalina.core.StandardServer.await A valid shutdown command was
> received via the shutdown port. Stopping the Server instance.
>
> 21-Nov-2019 23:24:29.844 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["http-nio-5080"]
>
> 21-Nov-2019 23:24:29.850 INFO [main]
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
> ["ajp-nio-8009"]
>
> 21-Nov-2019 23:24:29.856 INFO [main]
> org.apache.catalina.core.StandardService.stopInternal Stopping service
> [Catalina]
>
> [34mINFO [0;39m 11-21 23:24:29.884 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket extensions initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket core library initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy:
> org.wicketstuff.dashboard.DashboardContextInitializer@72690a1
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
>
> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
> [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
> (theme-default)
>
> 21-Nov-2019 23:24:29.886 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
> [172.17.0.1]:5701 is SHUTTING_DOWN
>
> 21-Nov-2019 23:24:29.891 INFO [hz.server-1.cached.thread-2]
> com.hazelcast.internal.partition.impl.MigrationManager.null
> [172.17.0.1]:5701 [dev] [3.12] Shutdown request of Member [172.17.0.1]:5701
> - 799babea-5e78-4cf9-a099-f4b1fb09b9c9 this is handled
>
> 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Shutting down connection manager...
>
> 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Shutting down node engine...
>
> 21-Nov-2019 23:24:29.979 INFO [main]
> com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12]
> Destroying node NodeExtension.
>
> 21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.instance.Node.null
> [172.17.0.1]:5701 [dev] [3.12] Hazelcast Shutdown is completed in 90 ms.
>
> 21-Nov-2019 23:24:29.980 INFO [main]
> com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
> [172.17.0.1]:5701 is SHUTDOWN
>
> 21-Nov-2019 23:24:30.023 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
> web
> application [Openmeetings] registered the JDBC driver
> [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
> application was stopped. To prevent a memory leak, the JDBC Driver has been
> forcibly unregistered.
>
> 21-Nov-2019 23:24:30.024 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
>
> java.base@11/java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThr
> eadEventExecutor.java:719)
>
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventEx
> ecutor.java:905)
>
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable
> .java:30)
>
> java.base@11/java.lang.Thread.run(Thread.java:834)
>
> 21-Nov-2019 23:24:30.025 WARNING [main]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [Openmeetings] appears to have started a thread named
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
> has failed to stop it. This is very likely to create a memory leak. Stack
> trace of thread:
>
> java.base@11/java.lang.Object.wait(Native Method)
>
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555
> )
>
>
>
>
>
> Thank you.
>
>
>
> Sincerely,
>
>
>
> Hemant K. Sabat
>
> <http://www.coscend.com/> www.Coscend.com
>
> ------------------------------------------------------------------
>
> Real-time, Interactive Video Collaboration, Tele-healthcare,
> Tele-education,
> Telepresence Services, on the fly.
>
> ------------------------------------------------------------------
>
> CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
> Messages from Coscend Communications Solutions' posted at:
> <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
>
>
>
>
>
>
> --
>
> WBR
> Maxim aka solomax
>
--
WBR
Maxim aka solomax
RE: OM5: Tomcat not shutting down due to OM5 method
Posted by "Coscend@OM" <OM...@Coscend.com>.
Further, if we do not run OM5, Tomcat shuts down properly. Environment is CentOS 8 (using docker) and MariaDB 10.3.x
From: Coscend@OM [mailto:OM.Insights@Coscend.com]
Sent: Friday, November 22, 2019 12:12 PM
To: user@openmeetings.apache.org
Subject: RE: OM5: Tomcat not shutting down due to OM5 method
Dear Maxim,
I am sending this to users, but we think one possibility (not confirmation) is this error is related to OM5 code [1]. There is a method in OM5 that is triggering memory leak while shutting down Tomcat. The most common reason is something hanging or delaying in Web application initializations [2]. This prevents Tomcat to properly shutdown [1, 2].
Or, is due to something else? (We are using OM5 binary from OM Web site. There are no customization or configuration changes.)
Here is the relevant log.
The key line in the log is: “The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak.”
------------------------
21-Nov-2019 23:24:30.024 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
21-Nov-2019 23:24:30.025 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
[1] https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005
[2] https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly…
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
From: Maxim Solodovnik [mailto:solomax666@gmail.com]
Sent: Friday, November 22, 2019 11:36 AM
To: dev <dev@openmeetings.apache.org <ma...@openmeetings.apache.org> >; OM.Insights@coscend.com <ma...@coscend.com>
Subject: Re: OM5: Tomcat not shutting down due to OM5 method
Please send these messages to users@
It is not about OM/OM_community development
shutdown works for me as expected
please provide additional details (if any)
On Fri, 22 Nov 2019 at 13:04, Coscend@OM <OM.Insights@coscend.com <ma...@coscend.com> > wrote:
Dear OM Community,
While shutting down OM5-Tomcat via shutdown.sh, Tomcat is not shutting down.
These links state some method in Webapp could be preventing the shutdown
process of Tomcat:
https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shu <https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005>
tdown-port-8005
https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listen <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1>
ing?noredirect=1
<https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-liste <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1&lq=1>
ning?noredirect=1&lq=1> &lq=1
Could anyone provide any guidance what we need to modify in OM5 code or any
other place to properly shutdown Tomcat?
Here is the log snippet. There are several more such errors while shutting
down.
------------------------------
21-Nov-2019 23:24:29.843 INFO [main]
org.apache.catalina.core.StandardServer.await A valid shutdown command was
received via the shutdown port. Stopping the Server instance.
21-Nov-2019 23:24:29.844 INFO [main]
org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
["http-nio-5080"]
21-Nov-2019 23:24:29.850 INFO [main]
org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler
["ajp-nio-8009"]
21-Nov-2019 23:24:29.856 INFO [main]
org.apache.catalina.core.StandardService.stopInternal Stopping service
[Catalina]
[34mINFO [0;39m 11-21 23:24:29.884 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket jQuery UI initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket extensions initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket core library initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy:
org.wicketstuff.dashboard.DashboardContextInitializer@72690a1 <ma...@72690a1>
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins)
[34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m -
[OpenmeetingsApplication] destroy: Wicket Kendo UI initializer
(theme-default)
21-Nov-2019 23:24:29.886 INFO [main]
com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
[172.17.0.1]:5701 is SHUTTING_DOWN
21-Nov-2019 23:24:29.891 INFO [hz.server-1.cached.thread-2]
com.hazelcast.internal.partition.impl.MigrationManager.null
[172.17.0.1]:5701 [dev] [3.12] Shutdown request of Member [172.17.0.1]:5701
- 799babea-5e78-4cf9-a099-f4b1fb09b9c9 this is handled
21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Shutting down connection manager...
21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Shutting down node engine...
21-Nov-2019 23:24:29.979 INFO [main]
com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12]
Destroying node NodeExtension.
21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.instance.Node.null
[172.17.0.1]:5701 [dev] [3.12] Hazelcast Shutdown is completed in 90 ms.
21-Nov-2019 23:24:29.980 INFO [main]
com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12]
[172.17.0.1]:5701 is SHUTDOWN
21-Nov-2019 23:24:30.023 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web
application [Openmeetings] registered the JDBC driver
[com.mysql.cj.jdbc.Driver] but failed to unregister it when the web
application was stopped. To prevent a memory leak, the JDBC Driver has been
forcibly unregistered.
21-Nov-2019 23:24:30.024 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [Openmeetings] appears to have started a thread named
[nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to
create a memory leak. Stack trace of thread:
java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method)
io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThr
eadEventExecutor.java:719)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventEx
ecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable
.java:30)
java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)>
21-Nov-2019 23:24:30.025 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [Openmeetings] appears to have started a thread named
[org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but
has failed to stop it. This is very likely to create a memory leak. Stack
trace of thread:
java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555
)
Thank you.
Sincerely,
Hemant K. Sabat
<http://www.coscend.com/> www.Coscend.com <http://www.Coscend.com>
------------------------------------------------------------------
Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education,
Telepresence Services, on the fly.
------------------------------------------------------------------
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
Messages from Coscend Communications Solutions' posted at:
<http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html>
http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html
--
WBR
Maxim aka solomax