[jira] [Created] (IMPALA-9858) Wrong partition hit/request metrics in profile of LocalCatalog

["Quanlong Huang (Jira)" <ji...@apache.org>]

Quanlong Huang created IMPALA-9858:
--------------------------------------

             Summary: Wrong partition hit/request metrics in profile of LocalCatalog
                 Key: IMPALA-9858
                 URL: https://issues.apache.org/jira/browse/IMPALA-9858
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
    Affects Versions: Impala 3.4.0, Impala 3.3.0, Impala 3.2.0, Impala 3.1.0
            Reporter: Quanlong Huang
            Assignee: Quanlong Huang


The LocalCatalog metrics of "CatalogFetch.Partitions.Hits" and "CatalogFetch.Partitions.Requests" in the profile are overcounted. For query "select * from functional.alltypes" where "functional.alltypes" contains 24 partitions, the partition metrics on a cold-started LocalCatalog coordinator are:
{code:java}
       - CatalogFetch.Partitions.Hits: 48
       - CatalogFetch.Partitions.Misses: 24
       - CatalogFetch.Partitions.Requests: 72{code}
Actually, only 48 requests are made on the local cache. 24 of them come from partition pruning and encounter cache miss. Another 24 of them come from LocalFsTable.toThriftDescriptor() and hit cache.

The overcounting is due to a bug at [https://github.com/apache/impala/blob/f8c28f8adfd781727c311b15546a532ce65881e0/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java#L898]

Code snipper:
{code:java}
  public Map<String, PartitionMetadata> loadPartitionsByRefs(TableMetaRef table,
    ......
    final int numHits = refToMeta.size();
    final int numMisses = partitionRefs.size() - numHits;

    // Load the remainder from the catalogd.
    List<PartitionRef> missingRefs = new ArrayList<>();
    for (PartitionRef ref: partitionRefs) {
      if (!refToMeta.containsKey(ref)) missingRefs.add(ref);
    }
    if (!missingRefs.isEmpty()) {
      Map<PartitionRef, PartitionMetadata> fromCatalogd = loadPartitionsFromCatalogd(
          refImpl, hostIndex, missingRefs);
      refToMeta.putAll(fromCatalogd);    // <---- refToMeta is updated here!
      // Write back to the cache.
      storePartitionsInCache(refImpl, hostIndex, fromCatalogd);
    }
    sw.stop();
    addStatsToProfile(PARTITIONS_STATS_CATEGORY, refToMeta.size(), numMisses, sw);   // <--- Should use numHits instead of refToMeta.size()
    LOG.trace("Request for partitions of {}: hit {}/{}", table, refToMeta.size(),
        partitionRefs.size());
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org