You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2025/03/04 19:49:00 UTC

[jira] [Updated] (HUDI-8909) Support/Fix Byte data type w/ partition stats

     [ https://issues.apache.org/jira/browse/HUDI-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sivabalan narayanan updated HUDI-8909:
--------------------------------------
    Remaining Estimate: 4h
     Original Estimate: 4h

> Support/Fix Byte data type w/ partition stats 
> ----------------------------------------------
>
>                 Key: HUDI-8909
>                 URL: https://issues.apache.org/jira/browse/HUDI-8909
>             Project: Apache Hudi
>          Issue Type: Sub-task
>          Components: metadata
>            Reporter: sivabalan narayanan
>            Assignee: sivabalan narayanan
>            Priority: Major
>             Fix For: 1.0.2
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> While working towards making partition stats default, we ran into an issue, with Byte data type 
> [https://github.com/apache/hudi/pull/12671]
>  
> min max values when merging multiple values did not align w/ manually computed stats. Check for column "c7" in tests in TestColStatsIndex. 
>  
> To reproduce:
> switch data type of C7 to "Byte". 
> and run 
> TestColumnStatsIndex.testMetadataColumnStatsIndex. 
>  
> comment out 
> ```
> assertEquals(asJson(sort(expectedColStatsIndexTableDf, validationSortColumns)),
> asJson(sort(transposedColStatsDF.drop("fileName"), validationSortColumns)))
> ```
> in ColumnStatIndexTestBase.
>  
> Run the test for COW table. you may find the issue w/ below validation
> ```
> assertEquals(asJson(sort(manualColStatsTableDF.drop(colsToDrop: _*), pValidationSortColumns)),
> asJson(sort(pTransposedColStatsDF.drop(colsToDrop: _*), pValidationSortColumns)))
> ```
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)