You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/01/13 01:09:01 UTC
[jira] [Commented] (IMPALA-9858) Wrong partition hit/request
metrics in profile of LocalCatalog
[ https://issues.apache.org/jira/browse/IMPALA-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263804#comment-17263804 ]
ASF subversion and git services commented on IMPALA-9858:
---------------------------------------------------------
Commit fe4de658209c94137beab1e477ac6fef5aa9d8db in impala's branch refs/heads/3.x from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=fe4de65 ]
IMPALA-9858: Fix wrong partition metrics in LocalCatalog profile
The hits and requests metrics of partitions are overcounted due to using
an updated map. This patch fixes it and adds test coverage on partition
metrics.
Tests
- Run CatalogdMetaProviderTest
Change-Id: I10cabce2908f1d252b90390978e679d31003e89d
Reviewed-on: http://gerrit.cloudera.org:8080/16080
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
(cherry picked from commit ee70df2e9006d3592175af5f1fa7ec128f5f1b8d)
> Wrong partition hit/request metrics in profile of LocalCatalog
> --------------------------------------------------------------
>
> Key: IMPALA-9858
> URL: https://issues.apache.org/jira/browse/IMPALA-9858
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.1.0, Impala 3.2.0, Impala 3.3.0, Impala 3.4.0
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
> Fix For: Impala 4.0
>
>
> The LocalCatalog metrics of "CatalogFetch.Partitions.Hits" and "CatalogFetch.Partitions.Requests" in the profile are overcounted. For query "select * from functional.alltypes" where "functional.alltypes" contains 24 partitions, the partition metrics on a cold-started LocalCatalog coordinator are:
> {code:java}
> - CatalogFetch.Partitions.Hits: 48
> - CatalogFetch.Partitions.Misses: 24
> - CatalogFetch.Partitions.Requests: 72{code}
> Actually, only 48 requests are made on the local cache. 24 of them come from partition pruning and encounter cache miss. Another 24 of them come from LocalFsTable.toThriftDescriptor() and hit cache.
> The overcounting is due to a bug at [https://github.com/apache/impala/blob/f8c28f8adfd781727c311b15546a532ce65881e0/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java#L898]
> Code snipper:
> {code:java}
> public Map<String, PartitionMetadata> loadPartitionsByRefs(TableMetaRef table,
> ......
> final int numHits = refToMeta.size();
> final int numMisses = partitionRefs.size() - numHits;
> // Load the remainder from the catalogd.
> List<PartitionRef> missingRefs = new ArrayList<>();
> for (PartitionRef ref: partitionRefs) {
> if (!refToMeta.containsKey(ref)) missingRefs.add(ref);
> }
> if (!missingRefs.isEmpty()) {
> Map<PartitionRef, PartitionMetadata> fromCatalogd = loadPartitionsFromCatalogd(
> refImpl, hostIndex, missingRefs);
> refToMeta.putAll(fromCatalogd); // <---- refToMeta is updated here!
> // Write back to the cache.
> storePartitionsInCache(refImpl, hostIndex, fromCatalogd);
> }
> sw.stop();
> addStatsToProfile(PARTITIONS_STATS_CATEGORY, refToMeta.size(), numMisses, sw); // <--- Should use numHits instead of refToMeta.size()
> LOG.trace("Request for partitions of {}: hit {}/{}", table, refToMeta.size(),
> partitionRefs.size());
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org