You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/12/19 12:11:45 UTC

[GitHub] [pulsar] lifepuzzlefun opened a new issue, #18985: PIP-232: Introduce thread monitor to check if thread is blocked for long time.

lifepuzzlefun opened a new issue, #18985:
URL: https://github.com/apache/pulsar/issues/18985

   ### Motivation
   
   As we know. Pulsar is a system written in asynchronous style. 
   In previous version, there were some cases which will cause thread blocking for a long time. 
   This PIP want to introduce some lightweight apm mechanism to check if thread is blocking for a long time.
   
   ### Goal
   
   1. monitor most of pulsar threads.
   2. low overhead.
   4. support dynamic enable/disable.
   
   ### API Changes
   
   no change of API.
   
   ### Implementation
   
   # Basic idea
   
   For single thread executor. there is a task queue for the thread. so if a task insert into the queue and executed. we think at least the thread is not blocking. Otherwise, the task will stay in the queue for very long time.
   
   So we can register the executor pool to the monitor class and submit `ping` task periodicly. if task is executed. we update the  
   thread `lastActiveTimestamp` and check if there were some threads whose active timestamp is not refreshed in time. Then we think perhaps the thread is blocking.
   
   For multi thread executor. Like `Executors.newScheduledThreadPool(100)` the task queue is shared among all the threads in the pool. so the `ping` task is not guaranteed to execute on different thread every time. and the `lastActiveTimestamp` is not refreshed in time. 
   
   To monitor those threads in the multi thread executor. each thread will have a `runningTask` state marker.
   if `lastActiveTimstamp` is not refreshed in time and `runningTask` is `false` we think the thread is just health to run task.
   
   JDK `ThreadPoolExecutor.beforeExecute(Thread t, Runnable r)` is called before each task submit in the task queue. and `ThreadPoolExecutor.afterExecute(Runnable r, Throwable t)` is called after task is finished. So we can override those methods to update `runningTask` state of the threads in pool.
   
   
   # Current Pulsar Thread Category
   
   1. Single thread executor with one queue bind on one thread.
   Netty `EventLoop` & `OrderedExecutor` & `OrderedScheduler` & `Executors. newSingleThreadScheduledExecutor`
   
   2. Multi thread executor with one queue bind mult-threads.
   `WebExecutorThreadPool` used by `pulsar-web` and `pulsar-websocket` and those created by `Executors.newXXXX`
   
   3. `ForkJoinPool.commonPool` used by `CompletableFuture` callbacks.
   
   # Changes
   ### `ThreadMonitor` and `ThreadPoolMonitor` will be responsible for the main logic. 
   
   #### 1. `ThreadMonitor` 
   will have a `Map<threadId, threadMonitorState{lastActiveTimestamp,runningTask}>` to record the monitor state.
   
   #### 2. `ThreadPoolMonitor` 
   1. will schedule periodicly `ping` tasks to all registered executor pool. 
   2. and also there were some logic about dynamic enable/disable/reconfiguration the monitor logic. Reconfiguration logic is running by schedule tasks to read the system property to check if the monitor config is changed. 
   3. check blocking thread and dump stack in the log. If the condition (`lastActiveTimestamp` not refresh in time and `runningTask` == false).Those thread stack will be dumped using ThreadMXBean provide by JDK.
   4. exit thread cleanup. a schedule task for cleaning exited thread recorded in `ThreadMonitor`.
   
   #### 3. `ExecutorProvider` and `ScheduledExecutorProvider`
   1. now all executor and schedule executor will be create by these class.
   eg. 
   `Executors.newXXXXX` -> `ExecutorProvider.newXXXX` 
   `Executors.newScheduleXXXX` -> `ScheduleredExecutorProvider.newXXXXX`
   
   2. those executors created by Provider will be registered in the `ThreadPoolMonitor` automatically.
   3. Also `EventLoopGroup` created by `EventLoopUtil` will be registered in the `ThreadPoolMonitor` automatically.
   
   # Metrics
   1. `pulsar_thread_last_active_timestamp_ms{threadName="${threadName},tid="${tid},runningTask="${runningTask}"}` will be output in the promethues metric endpoint.
   
   4. `thread_blocked_count`, `thread_blocked_time_ms`, `thread_waited_count`, `thread_waited_time_ms` will be output in the prometheus metric endpoint. if `TheadMXBean.isThreadContentionMonitoringSupported` return true. 
   > see `https://docs.oracle.com/javase/8/docs/api/java/lang/management/ThreadInfo.html` for more information. note that
   > those are counters and won't be reset when thread return from block or wait state. Also when the thread have no task to do
   > will also cause those counter to add up.
   
   # Configuration
   For current implementation all the config is set by system property. And there were some tools can change system property in the runtime.
   
   1. `pulsar.thread-pool.monitor.enabled` if schedule `ping` task to registerd executors.
   2. `pulsar.thread-pool.monitor.check.ms` interval to schedule `ping` task to registered executors.
   
   5. `pulsar.thread-pool.monitor.blocking.dump.enabled` if dump blocking threads.
   6. `pulsar.thread-pool.monitor.blocking.dump.check.ms` interval to check if there were thread `lastActiveTimestamp` not refreshed in time.
   7. `pulsar-thread-pool.monitor.blocking.dump.threshold.ms` >= this value will think thread is blocking.
   8. `pulsar-thread-pool.monitor.blocking.dump.stack-max-depth` thread max dump stack depth.
   
   
   # Left for discussion
   `ForkJoinPool` is not easy to use the method as mentioned above.
   1. Most CompletableFuture callback use `ForkJoinPool.commonPool` is not easy to replace or override callback method.
   2. If we implement our own `CompletableFuture` to trace each callback pending  in the queue. maybe this will introduce a lot overhead which will harm the performance.
   
   > see `https://stackoverflow.com/questions/61759182/monitoring-debugging-and-tracking-completablefuture-tasks` for more information.
   
   
   
   
   
   
    
   
   ### Alternatives
   
   _No response_
   
   ### Anything else?
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] PIP-232: Introduce thread monitor to check if thread is blocked for long time. [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #18985:
URL: https://github.com/apache/pulsar/issues/18985#issuecomment-6377557321

   Thanks @lifepuzzlefun for the proposal. PIP-232 was discussed on the dev@ list (https://lists.apache.org/thread/2hqn5k3l64vs1d0tjrxbzr292sfl4qtg), where reviewers asked for a less invasive approach (for example BlockHound, or starting with low-overhead metrics), but it never went to a vote and has had no activity since early 2023. I'm closing this issue as not planned. Anyone is welcome to restart it as a pip/ document following https://github.com/apache/pulsar/blob/master/pip/README.md.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] PIP-232: Introduce thread monitor to check if thread is blocked for long time. [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #18985: PIP-232: Introduce thread monitor to check if thread is blocked for long time.
URL: https://github.com/apache/pulsar/issues/18985


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] lhotari commented on issue #18985: PIP-232: Introduce thread monitor to check if thread is blocked for long time.

Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #18985:
URL: https://github.com/apache/pulsar/issues/18985#issuecomment-1368951197

   I replied on the mailing list thread: https://lists.apache.org/thread/p76y2p4cst06nmo8yzcw8pyg19x6rtnt


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] lhotari commented on issue #18985: PIP-232: Introduce thread monitor to check if thread is blocked for long time.

Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #18985:
URL: https://github.com/apache/pulsar/issues/18985#issuecomment-1368899072

   @lifepuzzlefun Have you considered integrating existing tooling such as [Reactor BlockHound](https://github.com/reactor/BlockHound) to detect blocking code that is run on threads which should be non-blocking?
   
   These BlockHound docs explains how to write an integration for BlockHound: https://github.com/reactor/BlockHound/blob/master/docs/custom_integrations.md and https://github.com/reactor/BlockHound/blob/master/docs/customization.md
   
   Essentially, it's about configuring which threads should be non-blocking and about white-listing some known violations. 
   
   For example, Netty already comes with a BlockHound integration: https://github.com/netty/netty/blob/f027fa2df77af7719aa9686659633ee0fc73ebf9/common/src/main/java/io/netty/util/internal/Hidden.java#L31-L186
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on issue #18985: PIP-232: Introduce thread monitor to check if thread is blocked for long time.

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #18985:
URL: https://github.com/apache/pulsar/issues/18985#issuecomment-1413043656

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org