You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@impala.apache.org by "jhkcool (Jira)" <ji...@apache.org> on 2022/12/23 02:10:00 UTC

[jira] [Created] (IMPALA-11810) The mem estimate is incorrect in HdfsScanNode

jhkcool created IMPALA-11810:
--------------------------------

             Summary: The mem estimate is incorrect in HdfsScanNode 
                 Key: IMPALA-11810
                 URL: https://issues.apache.org/jira/browse/IMPALA-11810
             Project: IMPALA
          Issue Type: Improvement
          Components: fe
    Affects Versions: Impala 4.1.1, Impala 4.2.0
            Reporter: jhkcool


About perInstanceMemEstimate calculate int below method:
{code:java}
@Override  
public void computeNodeResourceProfile(TQueryOptions queryOptions) {
...
    long avgScanRangeBytes = (long) Math.ceil(sumValues(totalBytesPerFs_) / (double) scanRangeSize);
...
}{code}
All table data file sizes are used in the calculation of scan hdfs memory consumption, it is unreasonable. Because not all data in the table needs to be read into the memory, but only the materialized fields are read, so only the data file size occupied by the materialized field needs to be calculated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)