You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Lars Volker (JIRA)" <ji...@apache.org> on 2018/11/29 21:45:00 UTC
[jira] [Updated] (IMPALA-6197) Instrument our locks to track
contention
[ https://issues.apache.org/jira/browse/IMPALA-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Volker updated IMPALA-6197:
--------------------------------
Description:
We should add instrumentation to our locks (or switch to an alternative like [absl::mutex|https://abseil.io/about/design/mutex]) to enable profiling of lock contention (see [absl::RegisterMutexProfiler()|https://github.com/abseil/abseil-cpp/blob/master/absl/synchronization/mutex.h#L933].
This isn't a fully baked idea but could be very helpful with diagnosing issues around lock contention and lock order.
[~tarmstrong] suggested, that it would be nice to be able to identify the following things:
* Highly contended locks, based on how often threads get blocked and how long they get blocked for. E.g. Vincent Tran was looking at an issue that looked like IMPALA-4456 but we had no way to confirm.
* Validation of lock order by defining a static lock order and then enforcing it with DCHECKs
* Detection of deadlocks or livelocks involving multiple locks. This requires a graph of which locks are held or waited for by the different threads.
* Some kind of per-query metrics for the profile, e.g. the number of times threads got blocked on locks.
* The difficulty is doing this with low overhead and integrating with the lock implementations that we currently use (boost::mutex, boost::shared_mutex and SpinLock). Maybe it's best to do it incrementally or selectively only for locks that we suspect will be contention points.
Abseil (Apache licensed) has some of this functionality: https://github.com/abseil/abseil-cpp/blob/master/absl/synchronization/mutex.cc
was:We should add instrumentation to our locks (or switch to an alternative like [absl::mutex|https://abseil.io/about/design/mutex]) to enable profiling of lock contention (see [absl::RegisterMutexProfiler()|https://github.com/abseil/abseil-cpp/blob/master/absl/synchronization/mutex.h#L933].
> Instrument our locks to track contention
> ----------------------------------------
>
> Key: IMPALA-6197
> URL: https://issues.apache.org/jira/browse/IMPALA-6197
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Lars Volker
> Priority: Major
> Labels: observability
>
> We should add instrumentation to our locks (or switch to an alternative like [absl::mutex|https://abseil.io/about/design/mutex]) to enable profiling of lock contention (see [absl::RegisterMutexProfiler()|https://github.com/abseil/abseil-cpp/blob/master/absl/synchronization/mutex.h#L933].
> This isn't a fully baked idea but could be very helpful with diagnosing issues around lock contention and lock order.
> [~tarmstrong] suggested, that it would be nice to be able to identify the following things:
> * Highly contended locks, based on how often threads get blocked and how long they get blocked for. E.g. Vincent Tran was looking at an issue that looked like IMPALA-4456 but we had no way to confirm.
> * Validation of lock order by defining a static lock order and then enforcing it with DCHECKs
> * Detection of deadlocks or livelocks involving multiple locks. This requires a graph of which locks are held or waited for by the different threads.
> * Some kind of per-query metrics for the profile, e.g. the number of times threads got blocked on locks.
> * The difficulty is doing this with low overhead and integrating with the lock implementations that we currently use (boost::mutex, boost::shared_mutex and SpinLock). Maybe it's best to do it incrementally or selectively only for locks that we suspect will be contention points.
> Abseil (Apache licensed) has some of this functionality: https://github.com/abseil/abseil-cpp/blob/master/absl/synchronization/mutex.cc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org