You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2024/07/24 16:44:34 UTC
[I] Add metric to count zombie scans threads in tablet server [accumulo]
keith-turner opened a new issue, #4756:
URL: https://github.com/apache/accumulo/issues/4756
**Is your feature request related to a problem? Please describe.**
In the situation where client starts a long running scan and then the client goes away before reading all of the results of the scan a best effort attempt is made to cancel the scan on the server side. The best effort attempt involves interrupting the thread [in this code](https://github.com/apache/accumulo/blob/f54a2b214ab5ac38452536d5c9fd992813b1ba45/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Scanner.java#L195-L221) that is called when the scan session [is closed](https://github.com/apache/accumulo/blob/f54a2b214ab5ac38452536d5c9fd992813b1ba45/server/tserver/src/main/java/org/apache/accumulo/tserver/session/SingleScanSession.java#L59-L73). Its possible that the scan thread keeps on running after being interrupted. The thread no longer shows up in list scans because that looks at the scan sessions and the scan session is removed after it makes it best effort cleanup attempt. These threads can cause problems with closing a tablet and continue to consume re
sources on the tablet server. Its possible that these zombie scan threads could consume all of the slot in a scan threads pool blocking any new scans from running on the tablet server.
**Describe the solution you'd like**
Nothing can be done about a scan thread that ignores attempts to interrupt it other than kill the process. Since these threads can cause serious problems over the long term it would be useful to have a new metric that provides a count of zombie scan threads running on a tablet server. This metric would make it easy to find tablet servers that are experiencing this problem.
**Describe alternatives you've considered**
Could possibly have a tablets server process kill itself if this condition persist too long. However it would probably be better to have this done externally using the new metric rather than baking this behavior in to the tablet server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add metric to count zombie scans threads in tablet server [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #4756:
URL: https://github.com/apache/accumulo/issues/4756#issuecomment-2251299058
> Is there a good way to determine which scan threads are running zombie scans?
Not that I know of, I think some new code would need to be written. We currently have Session objects that can get to a scan thread, but the session objects may be removed. Tablets have an activeScans set that may offer a way to find threads, however if #4757 is implemented then Tablet objects could be removed. So once the Tablet and Session are removed (no longer tracked by the tablet server) then a zombie scan thread would have nothing referencing it in the tablet server except for maybe the thread pool.
We could add a tablet server level tracking object like the following. Scans threads could add/remove their self in a try/finally block. Periodically we could all countZombieScans() and update a gauge w/ the count.
```java
// track the running scans in the tablet server
class RunningScans {
Set<ScanSession> runningScans = new HashSet<>();
// TODO could return closeable that does the removal, would work well w/ try with resources
public sync void addScan(ScanSession scanSession) {
runningScans.add(scanSession);
}
public sync void removeScan(ScanSession scanSession) {
runningScans.remove(scanSession);
}
public sync int countZombieScans(){
return runningScans.stream().filter(scanSessions->scanSession.state==REMOVED).count();
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add metric to count zombie scans threads in tablet server [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #4756:
URL: https://github.com/apache/accumulo/issues/4756#issuecomment-2250303108
Looking at the javadoc for `Thread.interrupt` it looks like there is a case where an InterruptedException (or some other type of exception) is not thrown, but the thread's interrupted state is set. I'm thinking that `SortedKeyValueIterator.hasTop`, `SortedKeyValueIterator.seek` and `SortedKeyValueIterator.next` should check this via `Thread.interrupted` and if the result is `true`, then it should throw an Exception. If we changed these SKVI interface methods to have a default implementation that performed these checks, I'm not sure that there is a way to force all subclasses to call the superclass method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add metric to count zombie scans threads in tablet server [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner closed issue #4756: Add metric to count zombie scans threads in tablet server
URL: https://github.com/apache/accumulo/issues/4756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add metric to count zombie scans threads in tablet server [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #4756:
URL: https://github.com/apache/accumulo/issues/4756#issuecomment-2250276640
Is there a good way to determine which scan threads are running zombie scans? Interrupt is called on the thread, but I'm not sure that interrupt in a normal manner kills the thread. I'm assuming that the thread can be re-used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org