You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/04/01 14:39:00 UTC
[jira] [Commented] (IMPALA-9473) Add counts of the number of hits,
misses, and cache entries
[ https://issues.apache.org/jira/browse/IMPALA-9473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072828#comment-17072828 ]
ASF subversion and git services commented on IMPALA-9473:
---------------------------------------------------------
Commit 5acce4d2000485b9cd87a545b2a05154c889181c in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5acce4d ]
IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache
This adds two sets of metrics. The first is per-partition metrics
to track the performance of the underlying filesystem for the
data cache. It keeps histograms of read, write, and eviction
latency for each data cache partition along with another metric
recording the path for the partition. These are exposed as the
following metrics:
impala-server.io-mgr.remote-data-cache-partition-$0.path
impala-server.io-mgr.remote-data-cache-partition-$0.read-latency
impala-server.io-mgr.remote-data-cache-partition-$0.write-latency
impala-server.io-mgr.remote-data-cache-partition-$0.eviction-latency
This also adds metrics to keep counts of hits, misses, and entries
in the data cache. Since reducing the latency of IO is an important
feature of the data cache, the absolute count of hits and misses
is as important as the hit bytes and miss bytes. This adds the
following metrics:
impala-server.io-mgr.remote-data-cache-hit-count
impala-server.io-mgr.remote-data-cache-miss-count
impala-server.io-mgr.remote-data-cache-num-entries
To track metrics around inserts, this also adds the following
metrics:
impala-server.io-mgr.remote-data-cache-num-inserts
impala-server.io-mgr.remote-data-cache-dropped-entries
impala-server.io-mgr.remote-data-cache-instant-evictions
An instant eviction happens when inserting an entry into the cache
fails and the entry is immediately evicted during insert. This is
currently only possible for LIRS when the entry's size is larger
than the unprotected capacity. This manifests when the cache
size is very small. For example, for an 8MB entry, this would
manifest when a cache shard is smaller than 160MB. This metric
is primarily for debugging.
Testing:
- Hand testing to verify the per-partition latency histograms
- Modified custom_cluster/test_data_cache.py to also test
the counts.
Change-Id: I56a57d75ff11f00ebc85b85bcaf104fb8108c478
Reviewed-on: http://gerrit.cloudera.org:8080/15382
Reviewed-by: Thomas Tauber-Marshall <tm...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Add counts of the number of hits, misses, and cache entries
> -----------------------------------------------------------
>
> Key: IMPALA-9473
> URL: https://issues.apache.org/jira/browse/IMPALA-9473
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 3.4.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> The data cache currently reports metrics about hit bytes, miss bytes, and total bytes in the cache. Since there are entries of a variety of sizes in the cache, it is useful to have the actual counts for hits, misses, and total number of entries in the cache.
> The cache improves performance more by reducing the latency of requests (proportional to # of hits) than by increasing the bandwidth of transfers (proportional to # of bytes). Cache performance may improve by preferring smaller entries and thus hit bytes. It is useful to track both types of metrics.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org