You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/07/29 07:33:16 UTC
[GitHub] [skywalking] boystray opened a new issue #5193: java.lang.OutOfMemoryError: Java heap space
boystray opened a new issue #5193:
URL: https://github.com/apache/skywalking/issues/5193
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [*] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
why happen OutOfMemoryError? after i setup skywalking , it will report OutOfMemoryError then close database,so report "Table not found " repeatly
___
### Bug
- Which version of SkyWalking, OS and JRE?
SkyWalking 7.0.0
OS centos
JRE java version "1.8.0_201"
- Which company or project?
- What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.
the following is log :
2020-07-28 00:10:29,429 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=46s390ms165µs522ns).
2020-07-28 00:11:31,458 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m27s892ms59µs253ns).
2020-07-28 00:13:01,647 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m31s852ms149µs819ns).
2020-07-28 00:13:58,866 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=52s710ms358µs61ns).
2020-07-28 00:15:50,978 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m6s346ms629µs613ns).
2020-07-28 00:17:17,712 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=2m5s581ms830µs94ns).
2020-07-28 00:19:09,280 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m47s879ms900µs736ns).
2020-07-28 00:22:17,587 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=2m53s786ms663µs590ns).
2020-07-28 00:24:46,488 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=2m29s268ms732µs70ns).
2020-07-28 00:26:29,663 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m35s575ms543µs221ns).
2020-07-28 00:29:24,010 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=2m39s809ms458µs759ns).
2020-07-28 00:31:59,025 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=2m53s387ms823µs928ns).
2020-07-28 00:32:57,759 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m27s722ms938µs807ns).
2020-07-28 00:35:23,575 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=56s134ms700µs105ns).
2020-07-28 00:36:06,795 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=58s804ms398µs297ns).
2020-07-28 00:37:21,913 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=56s657ms234µs568ns).
2020-07-28 00:39:08,709 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m54s88ms733µs513ns).
2020-07-28 00:40:10,867 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m2s299ms190µs742ns).
2020-07-28 00:40:56,785 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=56s944ms320µs926ns).
2020-07-28 00:42:17,899 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler - 70 [grpcServerPool-1-thread-15] ERROR [] - CANCELLED: cancelled before receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.26.0.jar:1.26.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275) [grpc-stub-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.26.0.jar:1.26.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:44:49,781 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=54s177ms771µs729ns).
2020-07-28 00:45:28,278 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=52s43ms72µs340ns).
2020-07-28 00:48:05,943 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=46s440ms226µs401ns).
2020-07-28 00:49:18,380 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m7s775ms831µs295ns).
2020-07-28 00:50:04,087 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=57s714ms452µs850ns).
2020-07-28 00:50:57,452 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler - 70 [grpcServerPool-1-thread-10] ERROR [] - CANCELLED: cancelled before receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.26.0.jar:1.26.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275) [grpc-stub-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.26.0.jar:1.26.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:52:06,622 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler - 70 [grpcServerPool-1-thread-8] ERROR [] - CANCELLED: cancelled before receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.26.0.jar:1.26.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275) [grpc-stub-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.26.0.jar:1.26.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:53:52,384 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler - 70 [grpcServerPool-1-thread-9] ERROR [] - CANCELLED: cancelled before receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.26.0.jar:1.26.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275) [grpc-stub-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.26.0.jar:1.26.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:55:01,390 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler - 70 [grpcServerPool-1-thread-16] ERROR [] - CANCELLED: cancelled before receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.26.0.jar:1.26.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275) [grpc-stub-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.26.0.jar:1.26.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:56:42,333 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler - 70 [grpcServerPool-1-thread-9] ERROR [] - CANCELLED: cancelled before receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.26.0.jar:1.26.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275) [grpc-stub-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.26.0.jar:1.26.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:58:15,634 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler - 70 [grpcServerPool-1-thread-17] ERROR [] - CANCELLED: cancelled before receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.26.0.jar:1.26.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275) [grpc-stub-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.26.0.jar:1.26.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 01:00:30,849 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 96 [pool-9-thread-1] WARN [] - History of segment delete failure
2020-07-28 01:00:56,985 - org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ServiceInstanceInventoryCacheDAO - 90 [pool-10-thread-1] ERROR [] - The database has been closed [90098-196]
org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException: The database has been closed [90098-196]
at org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient.executeQuery(JDBCHikariCPClient.java:124) ~[library-client-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ServiceInstanceInventoryCacheDAO.loadLastUpdate(H2ServiceInstanceInventoryCacheDAO.java:76) ~[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.updateServiceInstanceInventory(CacheUpdateTimer.java:101) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.update(CacheUpdateTimer.java:65) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.lambda$start$0(CacheUpdateTimer.java:56) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-7.0.0.jar:7.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_201]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_201]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed [90098-196]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:144) ~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Database.checkPowerOff(Database.java:524) ~[h2-1.4.196.jar:1.4.196]
at org.h2.command.Command.executeQuery(Command.java:227) ~[h2-1.4.196.jar:1.4.196]
at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:111) ~[h2-1.4.196.jar:1.4.196]
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) ~[HikariCP-3.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) ~[HikariCP-3.1.0.jar:?]
at org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient.executeQuery(JDBCHikariCPClient.java:115) ~[library-client-7.0.0.jar:7.0.0]
... 12 more
2020-07-28 01:05:54,986 - com.zaxxer.hikari.pool.HikariPool - 766 [HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=46s764ms522µs627ns).
2020-07-28 01:04:04,940 - org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2NetworkAddressInventoryCacheDAO - 81 [pool-10-thread-1] ERROR [] - HikariPool-1 - Connection is not available, request timed out after 39558ms.
org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException: HikariPool-1 - Connection is not available, request timed out after 39558ms.
at org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient.getConnection(JDBCHikariCPClient.java:73) ~[library-client-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient.getConnection(JDBCHikariCPClient.java:60) ~[library-client-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2NetworkAddressInventoryCacheDAO.loadLastUpdate(H2NetworkAddressInventoryCacheDAO.java:66) ~[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.updateNetAddressInventory(CacheUpdateTimer.java:125) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.update(CacheUpdateTimer.java:66) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.lambda$start$0(CacheUpdateTimer.java:56) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-7.0.0.jar:7.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_201]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_201]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 39558ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:676) ~[HikariCP-3.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:190) ~[HikariCP-3.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155) ~[HikariCP-3.1.0.jar:?]
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) ~[HikariCP-3.1.0.jar:?]
at org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient.getConnection(JDBCHikariCPClient.java:69) ~[library-client-7.0.0.jar:7.0.0]
... 13 more
2020-07-28 01:03:38,008 - org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO - 78 [DataCarrier.H2_ASYNCHRONOUS_BATCH_PERSISTENT.BulkConsumePool.0.Thread] ERROR [] - The database has been closed [90098-196]
org.h2.jdbc.JdbcSQLException: The database has been closed [90098-196]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:144) ~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getTransaction(Session.java:1636) ~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getStatementSavepoint(Session.java:1646) ~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.setSavepoint(Session.java:825) ~[h2-1.4.196.jar:1.4.196]
at org.h2.command.Command.executeUpdate(Command.java:254) ~[h2-1.4.196.jar:1.4.196]
at org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:207) ~[h2-1.4.196.jar:1.4.196]
at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) ~[HikariCP-3.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) ~[HikariCP-3.1.0.jar:?]
at org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLExecutor.invoke(SQLExecutor.java:53) ~[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO.synchronous(H2BatchDAO.java:75) [storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO$H2BatchConsumer.consume(H2BatchDAO.java:106) [storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:79) [apm-datacarrier-7.0.0.jar:7.0.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:50) [apm-datacarrier-7.0.0.jar:7.0.0]
2020-07-28 01:07:40,986 - org.apache.skywalking.oap.server.core.storage.PersistenceTimer - 122 [pool-8-thread-1] ERROR [] - Java heap space
java.lang.OutOfMemoryError: Java heap space
2020-07-28 01:07:40,986 - org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO - 78 [DataCarrier.H2_ASYNCHRONOUS_BATCH_PERSISTENT.BulkConsumePool.0.Thread] ERROR [] - The database has been closed [90098-196]
org.h2.jdbc.JdbcSQLException: The database has been closed [90098-196]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:144) ~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getTransaction(Session.java:1636) ~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getStatementSavepoint(Session.java:1646) ~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.setSavepoint(Session.java:825) ~[h2-1.4.196.jar:1.4.196]
at org.h2.command.Command.executeUpdate(Command.java:254) ~[h2-1.4.196.jar:1.4.196]
at org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:207) ~[h2-1.4.196.jar:1.4.196]
at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) ~[HikariCP-3.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) ~[HikariCP-3.1.0.jar:?]
at org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLExecutor.invoke(SQLExecutor.java:53) ~[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO.synchronous(H2BatchDAO.java:75) [storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO$H2BatchConsumer.consume(H2BatchDAO.java:106) [storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:79) [apm-datacarrier-7.0.0.jar:7.0.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:50) [apm-datacarrier-7.0.0.jar:7.0.0]
2020-07-28 01:06:53,781 - org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager - 147 [pool-3-thread-1] ERROR [] - Java heap space
java.lang.OutOfMemoryError: Java heap space
2020-07-28 01:07:40,988 - org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO - 78 [DataCarrier.H2_ASYNCHRONOUS_BATCH_PERSISTENT.BulkConsumePool.0.Thread] ERROR [] - The database has been closed [90098-196]
org.h2.jdbc.JdbcSQLException: The database has been closed [90098-196]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:144) ~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getTransaction(Session.java:1636) ~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getStatementSavepoint(Session.java:1646) ~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.setSavepoint(Session.java:825) ~[h2-1.4.196.jar:1.4.196]
at org.h2.command.Command.executeUpdate(Command.java:254) ~[h2-1.4.196.jar:1.4.196]
at org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:207) ~[h2-1.4.196.jar:1.4.196]
at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) ~[HikariCP-3.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) ~[HikariCP-3.1.0.jar:?]
at org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLExecutor.invoke(SQLExecutor.java:53) ~[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO.synchronous(H2BatchDAO.java:75) [storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO$H2BatchConsumer.consume(H2BatchDAO.java:106) [storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:79) [apm-datacarrier-7.0.0.jar:7.0.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:50) [apm-datacarrier-7.0.0.jar:7.0.0]
___
### Requirement or improvement
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #5193: java.lang.OutOfMemoryError: Java heap space
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #5193:
URL: https://github.com/apache/skywalking/issues/5193#issuecomment-666054960
You are talking about OOM of backend size? Of course, it will be OOM, we indicate very clearly, H2 is for demonstration. It is running in the memory mode, it must be OOM in some time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] boystray commented on issue #5193: java.lang.OutOfMemoryError: Java heap space
Posted by GitBox <gi...@apache.org>.
boystray commented on issue #5193:
URL: https://github.com/apache/skywalking/issues/5193#issuecomment-666053882
java OOM is occured again and dump is taken.
throught eclipse memory analyzer, the leak roport "One instance of "org.h2.mvstore.MVMap" loaded by "sun.misc.Launcher$AppClassLoader @ 0xe00cd650" occupies 219,162,096 (47.12%) bytes. The memory is accumulated in one instance of "org.h2.mvstore.MVMap" loaded by "sun.misc.Launcher$AppClassLoader @ 0xe00cd650".
it seems like a bug in H2 storage,so I change backend storage from H2 to mysql.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #5193: java.lang.OutOfMemoryError: Java heap space
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #5193:
URL: https://github.com/apache/skywalking/issues/5193
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #5193: java.lang.OutOfMemoryError: Java heap space
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #5193:
URL: https://github.com/apache/skywalking/issues/5193#issuecomment-665416450
Could you find which object(s) cause this OOM?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org