You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/02/12 07:03:00 UTC
[jira] [Commented] (IMPALA-11107) Allow specifying footer size in HdfsScanner::IssueFooterRanges
[ https://issues.apache.org/jira/browse/IMPALA-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17491272#comment-17491272 ]
ASF subversion and git services commented on IMPALA-11107:
----------------------------------------------------------
Commit bb23b4376239674117f2ebd5893b2c1ec8f321da in impala's branch refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=bb23b43 ]
IMPALA-11107: Allow specifying footer size in IssueFooterRanges
FOOTER_SIZE was a constant of 100KB in HdfsScanner::IssueFooterRanges,
an estimate based on Parquet format. Other scanner subclasses such as
HdfsOrcScanner expect that footer size is much lower at
16KB (orc::DIRECTORY_SIZE_GUESS in ORC lib). This patch adds
footer_size_estimate as a parameter so different file formats can ask
for different footer range sizes. Having a more precise initial range
can also help reduce waste in the data cache.
Testing:
- Pass core tests.
- Manually verify that 'IoReadSkippedBytes' reduced after this patch.
Change-Id: Ib0a5aab48324bf9e78fb0bb0bf6f649d87e89dfc
Reviewed-on: http://gerrit.cloudera.org:8080/18220
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Allow specifying footer size in HdfsScanner::IssueFooterRanges
> --------------------------------------------------------------
>
> Key: IMPALA-11107
> URL: https://issues.apache.org/jira/browse/IMPALA-11107
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 4.0.0
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Major
>
> Currently, FOOTER_SIZE is a constant of 100KB in HdfsScanner::IssueFooterRanges:
> [https://github.com/apache/impala/blob/57982ef/be/src/exec/hdfs-scanner.cc#L832]
> Some scanner subclass such as HdfsOrcScanner expect that footer size is much lower at 16KB. We should add footer_size as parameter so different file format can ask for different footer range size.
> Having more precise initial range can also help reduce waste in the data cache:
> [https://github.com/apache/impala/blob/57982ef/be/src/runtime/io/data-cache.h#L73]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org