[jira] [Commented] (IMPALA-10433) Use Iceberg's fixed partition transforms

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17347245#comment-17347245 ] 

ASF subversion and git services commented on IMPALA-10433:
----------------------------------------------------------

Commit 824b39e82967261f5a40f26318cf0913751b5134 in impala's branch refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=824b39e ]

IMPALA-10433: Use Iceberg's fixed partition transforms

Because of an Iceberg bug Impala didn't push predicates to
Iceberg for dates/timestamps when the predicate referred to a
value before the UNIX epoch.

https://github.com/apache/iceberg/pull/1981 fixed the Iceberg
bug, and lately Impala switched to an Iceberg version that has
the fix, therefore this patch enables predicate pushdown for all
timestamp/date values.

The above Iceberg patch maintains backward compatibility with the
old, wrong behavior. Therefore sometimes we need to read plus one
Iceberg partition than necessary.

Testing:
 * Updated current e2e tests

Change-Id: Ie67f41a53f21c7bdb8449ca0d27746158be7675a
Reviewed-on: http://gerrit.cloudera.org:8080/17417
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Use Iceberg's fixed partition transforms
> ----------------------------------------
>
>                 Key: IMPALA-10433
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10433
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Frontend
>            Reporter: Zoltán Borók-Nagy
>            Priority: Major
>              Labels: impala-iceberg
>
> Currently the Iceberg time and date partition transforms are wrong if the data is before the epoch.
> There's already an Iceberg pull request about it: [https://github.com/apache/iceberg/pull/1981]
> Becauce of this bug Impala doesn't prune Iceberg partitions if the predicate refers to timestamps before the epoch.
> Once the above pull request is merged we need to update our Iceberg dependency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org