You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Sagar Sumit (Jira)" <ji...@apache.org> on 2023/03/27 05:35:00 UTC
[jira] [Commented] (HUDI-5989) Fix the date convert issue when lazy fetching partition path & file slice for HoodieFileIndex is used
[ https://issues.apache.org/jira/browse/HUDI-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705187#comment-17705187 ]
Sagar Sumit commented on HUDI-5989:
-----------------------------------
[~voonhous] [~Bone An] Can you please add the stacktrace and link the fix?
> Fix the date convert issue when lazy fetching partition path & file slice for HoodieFileIndex is used
> -----------------------------------------------------------------------------------------------------
>
> Key: HUDI-5989
> URL: https://issues.apache.org/jira/browse/HUDI-5989
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: voon
> Assignee: voon
> Priority: Major
>
> For partition filter like date column, spark internally will convert it to day numbers after timestamp 0, but hudi lazy fetching doesn't respect this behavior, this pr tries to fix it.
>
> {code:java}
> INFO DataSourceStrategy: Pruning directories with: isnotnull(country#80), isnotnull(date#81),(country#80 = ID),(date#81=19415){code}
>
> This will fix the partition pruning bug introduced in: [https://github.com/apache/hudi/pull/6680]
> h3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)