You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by GitBox <gi...@apache.org> on 2021/02/10 17:28:55 UTC
[GitHub] [jackrabbit-oak] mbaedke opened a new pull request #270: OAK-9351: Optionally log thread dump on lease update failure
mbaedke opened a new pull request #270:
URL: https://github.com/apache/jackrabbit-oak/pull/270
Implemented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] dcokan commented on a change in pull request #270: OAK-9351: Optionally log thread dump on lease update failure
Posted by GitBox <gi...@apache.org>.
dcokan commented on a change in pull request #270:
URL: https://github.com/apache/jackrabbit-oak/pull/270#discussion_r574646131
##########
File path: oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java
##########
@@ -466,6 +471,19 @@ private void registerNodeStore() throws IOException {
@Override
public void handleLeaseFailure() {
Review comment:
Yeah I confirm. The log we can see is "stopped" but actually your code will be placed before "stopping oak" log
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mbaedke commented on a change in pull request #270: OAK-9351: Optionally log thread dump on lease update failure
Posted by GitBox <gi...@apache.org>.
mbaedke commented on a change in pull request #270:
URL: https://github.com/apache/jackrabbit-oak/pull/270#discussion_r574416231
##########
File path: oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java
##########
@@ -466,6 +471,19 @@ private void registerNodeStore() throws IOException {
@Override
public void handleLeaseFailure() {
Review comment:
Hi @dcokan, as far as I can see, this is invoked immediately after the lease failure is detected. Maybe I'm missing something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] dcokan commented on a change in pull request #270: OAK-9351: Optionally log thread dump on lease update failure
Posted by GitBox <gi...@apache.org>.
dcokan commented on a change in pull request #270:
URL: https://github.com/apache/jackrabbit-oak/pull/270#discussion_r574435921
##########
File path: oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java
##########
@@ -466,6 +471,19 @@ private void registerNodeStore() throws IOException {
@Override
public void handleLeaseFailure() {
Review comment:
`[2021-02-04 17:14:41.014] ERROR
oak-scheduled-executor-1733 org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo
[User:] This oak instance failed to update the lease in time and can therefore no longer access this DocumentNodeStore.
[2021-02-04 17:14:41.014] INFO
LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate
[User:] [async] [CLOSED OK] Async indexing run completed. Closing it now
[2021-02-04 17:14:41.019] DEBUG
LeaseFailureHandler-Thread o.a.jackrabbit.oak.plugins.atomic.AtomicCounterEditorProvider
[User:] Shutting down ScheduledExecutorService
[2021-02-04 17:14:41.111] INFO
LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore
[User:] Starting disposal of DocumentNodeStore with clusterNodeId: 1 (id: 1, startTime: 1612424247296, machineId: mac:005056a737fb, instanceId: /opt/servers/instance, pid: 90769, uuid: 2a6a5fa5-3378-4cb0-b5c1-328a95ebb557, readWriteMode: null, state: ACTIVE, revLock: NONE, oakVersion: 1.6.17)
[2021-02-04 17:14:41.111] INFO
e background read thread (1) org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore
[User:] Background operation BackgroundReadOperation successful again
[2021-02-04 17:14:41.111] INFO
background update thread (1) org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore
[User:] Background operation BackgroundOperation successful again
[2021-02-04 17:14:41.113] ERROR
LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo
[User:] This oak instance failed to update the lease in time and can therefore no longer access this DocumentNodeStore.
[2021-02-04 17:14:41.114] DEBUG
re cluster update thread (1) o.apache.jackrabbit.oak.plugins.document.DocumentStoreStats.perf
[User:] query for children from [0] to [a], lock:-1 [took 2ms]
[2021-02-04 17:14:41.179] INFO
LeaseFailureHandler-Thread o.a.jackrabbit.oak.plugins.document.DocumentNodeStoreService
[User:] Unregistering DocumentNodeStore because blobDataSource [com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource@2a788117] was unbound
[2021-02-04 17:14:41.179] INFO
LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore
[User:] Starting disposal of DocumentNodeStore with clusterNodeId: 1 (id: 1, startTime: 1612424247296, machineId: mac:005056a737fb, instanceId: /opt//servers/instance, pid: 90769, uuid: 2a6a5fa5-3378-4cb0-b5c1-328a95ebb557, readWriteMode: null, state: ACTIVE, revLock: NONE, oakVersion: 1.6.17)
[2021-02-04 17:14:41.179] INFO
LeaseFailureHandler-Thread o.a.jackrabbit.oak.plugins.document.DocumentNodeStoreService
[User:] Unregistering DocumentNodeStore because dataSource [com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource@2a788117] was unbound
[2021-02-04 17:14:41.281] DEBUG
LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.NRTIndex
[User:] [/oak:index/tmp:d_index] Closing NRTIndex [UNKNOWN]
[2021-02-04 17:14:41.281] DEBUG
LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.NRTIndex
[User:] [/oak:index/tmp:f_index] Closing NRTIndex [UNKNOWN]
[2021-02-04 17:14:41.281] DEBUG
LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.NRTIndex
[User:] [/oak:index/tmp:c_index] Closing NRTIndex [UNKNOWN]
[2021-02-04 17:14:41.311] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Trying to unregister the SecurityProvider...
[2021-02-04 17:14:41.311] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Aborting: no SecurityProvider is registered
[2021-02-04 17:14:41.311] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Trying to unregister the SecurityProvider...
[2021-02-04 17:14:41.311] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Aborting: no SecurityProvider is registered
[2021-02-04 17:14:41.311] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Trying to unregister the SecurityProvider...
[2021-02-04 17:14:41.311] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Aborting: no SecurityProvider is registered
[2021-02-04 17:14:41.317] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Trying to unregister the SecurityProvider...
[2021-02-04 17:14:41.317] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Aborting: no SecurityProvider is registered
[2021-02-04 17:14:41.318] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Trying to unregister the SecurityProvider...
[2021-02-04 17:14:41.318] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Aborting: no SecurityProvider is registered
[2021-02-04 17:14:41.318] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Trying to unregister the SecurityProvider...
[2021-02-04 17:14:41.318] INFO
LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
[User:] Aborting: no SecurityProvider is registered
[2021-02-04 17:14:45.966] INFO
Service-BackgroundWorker-[1] o.a.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteService
[User:] BackgroundWorker.run: end {finally}
[2021-02-04 17:15:13.062] ERROR
LeaseFailureHandler-Thread o.a.jackrabbit.oak.plugins.document.DocumentNodeStoreService
[User:] handleLeaseFailure: stopped oak-core. `
I am referring to above logs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mbaedke closed pull request #270: OAK-9351: Optionally log thread dump on lease update failure
Posted by GitBox <gi...@apache.org>.
mbaedke closed pull request #270:
URL: https://github.com/apache/jackrabbit-oak/pull/270
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] dcokan commented on a change in pull request #270: OAK-9351: Optionally log thread dump on lease update failure
Posted by GitBox <gi...@apache.org>.
dcokan commented on a change in pull request #270:
URL: https://github.com/apache/jackrabbit-oak/pull/270#discussion_r574646658
##########
File path: oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java
##########
@@ -466,6 +471,19 @@ private void registerNodeStore() throws IOException {
@Override
public void handleLeaseFailure() {
+ if (LOG.isDebugEnabled()) {
+ boolean createThreadDump = System.getProperty(
Review comment:
just for my reference why not use `Boolean.getBoolean("...")` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] dcokan commented on a change in pull request #270: OAK-9351: Optionally log thread dump on lease update failure
Posted by GitBox <gi...@apache.org>.
dcokan commented on a change in pull request #270:
URL: https://github.com/apache/jackrabbit-oak/pull/270#discussion_r574371022
##########
File path: oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java
##########
@@ -466,6 +471,19 @@ private void registerNodeStore() throws IOException {
@Override
public void handleLeaseFailure() {
Review comment:
Hi @mbaedke this is invoked post the datasource is unbound from the DocumentNodeStore. My point is it seems that this handler callback is invoked after the clean up already started. What if we would invoke it in `ClusterNodeInfo.performLeaseCheck` when it notices the slowness?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mbaedke commented on a change in pull request #270: OAK-9351: Optionally log thread dump on lease update failure
Posted by GitBox <gi...@apache.org>.
mbaedke commented on a change in pull request #270:
URL: https://github.com/apache/jackrabbit-oak/pull/270#discussion_r574451668
##########
File path: oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java
##########
@@ -466,6 +471,19 @@ private void registerNodeStore() throws IOException {
@Override
public void handleLeaseFailure() {
Review comment:
That looks fine to me. The shutdown is initiated from the LeaseFailureHandler _after_ the creation of the thread dump.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] dcokan commented on a change in pull request #270: OAK-9351: Optionally log thread dump on lease update failure
Posted by GitBox <gi...@apache.org>.
dcokan commented on a change in pull request #270:
URL: https://github.com/apache/jackrabbit-oak/pull/270#discussion_r574435921
##########
File path: oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java
##########
@@ -466,6 +471,19 @@ private void registerNodeStore() throws IOException {
@Override
public void handleLeaseFailure() {
Review comment:
> [2021-02-04 17:14:41.014] ERROR
> oak-scheduled-executor-1733 org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo
> [User:] This oak instance failed to update the lease in time and can therefore no longer access this DocumentNodeStore.
> [2021-02-04 17:14:41.014] INFO
> LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate
> [User:] [async] [CLOSED OK] Async indexing run completed. Closing it now
> [2021-02-04 17:14:41.019] DEBUG
> LeaseFailureHandler-Thread o.a.jackrabbit.oak.plugins.atomic.AtomicCounterEditorProvider
> [User:] Shutting down ScheduledExecutorService
> [2021-02-04 17:14:41.111] INFO
> LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore
> [User:] Starting disposal of DocumentNodeStore with clusterNodeId: 1 (id: 1, startTime: 1612424247296, machineId: mac:005056a737fb, instanceId: /opt/servers/instance, pid: 90769, uuid: 2a6a5fa5-3378-4cb0-b5c1-328a95ebb557, readWriteMode: null, state: ACTIVE, revLock: NONE, oakVersion: 1.6.17)
> [2021-02-04 17:14:41.111] INFO
> e background read thread (1) org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore
> [User:] Background operation BackgroundReadOperation successful again
> [2021-02-04 17:14:41.111] INFO
> background update thread (1) org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore
> [User:] Background operation BackgroundOperation successful again
> [2021-02-04 17:14:41.113] ERROR
> LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo
> [User:] This oak instance failed to update the lease in time and can therefore no longer access this DocumentNodeStore.
> [2021-02-04 17:14:41.114] DEBUG
> re cluster update thread (1) o.apache.jackrabbit.oak.plugins.document.DocumentStoreStats.perf
> [User:] query for children from [0] to [a], lock:-1 [took 2ms]
> [2021-02-04 17:14:41.179] INFO
> LeaseFailureHandler-Thread o.a.jackrabbit.oak.plugins.document.DocumentNodeStoreService
> [User:] Unregistering DocumentNodeStore because blobDataSource [com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource@2a788117] was unbound
> [2021-02-04 17:14:41.179] INFO
> LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore
> [User:] Starting disposal of DocumentNodeStore with clusterNodeId: 1 (id: 1, startTime: 1612424247296, machineId: mac:005056a737fb, instanceId: /opt//servers/instance, pid: 90769, uuid: 2a6a5fa5-3378-4cb0-b5c1-328a95ebb557, readWriteMode: null, state: ACTIVE, revLock: NONE, oakVersion: 1.6.17)
> [2021-02-04 17:14:41.179] INFO
> LeaseFailureHandler-Thread o.a.jackrabbit.oak.plugins.document.DocumentNodeStoreService
> [User:] Unregistering DocumentNodeStore because dataSource [com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource@2a788117] was unbound
> [2021-02-04 17:14:41.281] DEBUG
> LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.NRTIndex
> [User:] [/oak:index/tmp:d_index] Closing NRTIndex [UNKNOWN]
> [2021-02-04 17:14:41.281] DEBUG
> LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.NRTIndex
> [User:] [/oak:index/tmp:f_index] Closing NRTIndex [UNKNOWN]
> [2021-02-04 17:14:41.281] DEBUG
> LeaseFailureHandler-Thread org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.NRTIndex
> [User:] [/oak:index/tmp:c_index] Closing NRTIndex [UNKNOWN]
> [2021-02-04 17:14:41.311] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Trying to unregister the SecurityProvider...
> [2021-02-04 17:14:41.311] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Aborting: no SecurityProvider is registered
> [2021-02-04 17:14:41.311] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Trying to unregister the SecurityProvider...
> [2021-02-04 17:14:41.311] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Aborting: no SecurityProvider is registered
> [2021-02-04 17:14:41.311] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Trying to unregister the SecurityProvider...
> [2021-02-04 17:14:41.311] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Aborting: no SecurityProvider is registered
> [2021-02-04 17:14:41.317] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Trying to unregister the SecurityProvider...
> [2021-02-04 17:14:41.317] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Aborting: no SecurityProvider is registered
> [2021-02-04 17:14:41.318] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Trying to unregister the SecurityProvider...
> [2021-02-04 17:14:41.318] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Aborting: no SecurityProvider is registered
> [2021-02-04 17:14:41.318] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Trying to unregister the SecurityProvider...
> [2021-02-04 17:14:41.318] INFO
> LeaseFailureHandler-Thread o.a.j.oak.security.internal.SecurityProviderRegistration
> [User:] Aborting: no SecurityProvider is registered
> [2021-02-04 17:14:45.966] INFO
> Service-BackgroundWorker-[1] o.a.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteService
> [User:] BackgroundWorker.run: end {finally}
> [2021-02-04 17:15:13.062] ERROR
> LeaseFailureHandler-Thread o.a.jackrabbit.oak.plugins.document.DocumentNodeStoreService
> [User:] handleLeaseFailure: stopped oak-core.
I am referring to above logs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org