[jira] [Commented] (CASSANDRA-11713) Add ability to log thread dump when NTR pool is blocked

["Jeff Jirsa (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271910#comment-15271910 ] 

Jeff Jirsa commented on CASSANDRA-11713:
----------------------------------------

I know this sounds weird, but we've seen clusters that actually LOOK like they're behaving DESPITE the high NTR blocks, and I suspect a lot of unsuspecting users who would have been happy with that type of situation would be hurt severely by this patch because the log output in those clusters would be significant - for example, we've had 2-3 hour long stress runs where individual nodes have 7 digit ntr-blocked counters, can you imagine dumping threads to logs a million times an hour?.

At the very least, I'd personally like to see it guarded with a system property or log spam protection. As an operator, I'd strongly suggest we not commit this patch until one of those were available.

> Add ability to log thread dump when NTR pool is blocked
> -------------------------------------------------------
>
>                 Key: CASSANDRA-11713
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Observability
>            Reporter: Paulo Motta
>            Assignee: Paulo Motta
>            Priority: Minor
>
> Thread dumps are very useful for troubleshooting Native-Transport-Requests contention issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the conditions are transient and it's hard to catch the exact moment when they happen, so it could be useful to generate and log them upon user request when certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} that when enabled via JMX generates and logs a single thread dump on the system log when the thread pool queue is full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)