[jira] [Created] (HUDI-8037) Partition query for transformed value incorrectly prunes valid partitions

["Jonathan Vexler (Jira)" <ji...@apache.org>]

Jonathan Vexler created HUDI-8037:
-------------------------------------

             Summary: Partition query for transformed value incorrectly prunes valid partitions
                 Key: HUDI-8037
                 URL: https://issues.apache.org/jira/browse/HUDI-8037
             Project: Apache Hudi
          Issue Type: Bug
          Components: spark, spark-sql
            Reporter: Jonathan Vexler
            Assignee: Jonathan Vexler
             Fix For: 0.15.1


With timestamp keygen you can have a partition column with timestamps, but then use the keygen so it will create partitions based on days so that all records that have a timestamp on 7-31-2024 will go to the same parititon even though the values in the partition column differ by hours and minutes etc.

This causes a problem with partition pruning. lets say you query "select * from table where partition < 7-31-2024 at 7am and partition > 7-31-2024 at 6am ". Since the file structure has the partition of just 7-31-2024, that will be interpreted as 7-31-2024 at 12am. So the partition will be pruned from the search space.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)