[jira] [Created] (IMPALA-11278) Cardinality of small HBase regions is overestimated since HBASE-26340

["Csaba Ringhofer (Jira)" <ji...@apache.org>]

Csaba Ringhofer created IMPALA-11278:
----------------------------------------

             Summary: Cardinality of small HBase regions is overestimated since HBASE-26340
                 Key: IMPALA-11278
                 URL: https://issues.apache.org/jira/browse/IMPALA-11278
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog, Frontend
    Affects Versions: Impala 4.1.0
            Reporter: Csaba Ringhofer


Impala uses the size of an HBase region to estimate the number of rows, and the API we use (https://hbase.apache.org/2.4/apidocs/org/apache/hadoop/hbase/RegionLoad.html#getStorefileSizeMB() ) returns a size at MB precision. Since HBASE-26340 it returns 1 instead of 0 for very small but not empty tables, which leads to massively overestimating its size (we handle 0 in a special way. so we didn't estimate  row count as 0: https://github.com/apache/impala/blob/78609dca32d8ce996247c9552ba676a853c74686/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java#L585 )

In newer versions of HBase getStorefileSizeMB() is deprecated and there are functions to get the size at byte granulity. Using it could solve the massive overestimation, but it may make our planner tests more sensitive to small size changes in HBase regions.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org