You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/03/27 23:02:00 UTC
[jira] [Commented] (IMPALA-8834) Investigate enabling safe version of OPTIMIZE_PARTITION_KEY_SCANS by default
[ https://issues.apache.org/jira/browse/IMPALA-8834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705724#comment-17705724 ]
ASF subversion and git services commented on IMPALA-8834:
---------------------------------------------------------
Commit 794eb1ba4a6d459379dee91c4274be3f40bd16ac in impala's branch refs/heads/branch-4.1.2 from zhangyifan27
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=794eb1ba4 ]
IMPALA-11081: Fix incorrect results in partition key scan
This patch fixes incorrect results caused by short-circuit partition
key scan in the case where a Parquet/ORC file contains multiple
blocks.
IMPALA-8834 introduced the optimization that generating only one
scan range that corresponding to the first block per file. Backends
only issue footer ranges for Parquet/ORC files for file-metadata-only
queries(see HdfsScanner::IssueFooterRanges()), which leads to
incorrect results if the first block doesn't include a file footer.
This bug is fixed by returning a scan range corresponding to the last
block for Parquet/ORC files to make sure it contains a file footer.
Testing:
- Added e2e tests to verify the fix.
Backport Notes:
- Trivial conflicts in HdfsScanNode.java and test_partition_metadata.py
Change-Id: I17331ed6c26a747e0509dcbaf427cd52808943b1
Reviewed-on: http://gerrit.cloudera.org:8080/19471
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Investigate enabling safe version of OPTIMIZE_PARTITION_KEY_SCANS by default
> ----------------------------------------------------------------------------
>
> Key: IMPALA-8834
> URL: https://issues.apache.org/jira/browse/IMPALA-8834
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: perf
> Fix For: Impala 4.0.0
>
>
> Just an idea I had while updating the docs. We already have the logic in the planner to determine when a partition key scan has "distinct" semantics - i.e. you obtain the correct results so long as the scan returns 0 rows for a partition when there are no rows present, and at least one row when there is a row present (but the exact number doesn't affect correctness).
> We could push this knowledge down into the scan nodes and have them terminate early. This would be quite efficient if file handles and footers were already cached and greatly reduce the number of rows flowing through the rest of the plan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org