You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2025/03/04 19:49:00 UTC
[jira] [Updated] (HUDI-8909) Support/Fix Byte data type w/ partition stats
[ https://issues.apache.org/jira/browse/HUDI-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan updated HUDI-8909:
--------------------------------------
Remaining Estimate: 4h
Original Estimate: 4h
> Support/Fix Byte data type w/ partition stats
> ----------------------------------------------
>
> Key: HUDI-8909
> URL: https://issues.apache.org/jira/browse/HUDI-8909
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: metadata
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Major
> Fix For: 1.0.2
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> While working towards making partition stats default, we ran into an issue, with Byte data type
> [https://github.com/apache/hudi/pull/12671]
>
> min max values when merging multiple values did not align w/ manually computed stats. Check for column "c7" in tests in TestColStatsIndex.
>
> To reproduce:
> switch data type of C7 to "Byte".
> and run
> TestColumnStatsIndex.testMetadataColumnStatsIndex.
>
> comment out
> ```
> assertEquals(asJson(sort(expectedColStatsIndexTableDf, validationSortColumns)),
> asJson(sort(transposedColStatsDF.drop("fileName"), validationSortColumns)))
> ```
> in ColumnStatIndexTestBase.
>
> Run the test for COW table. you may find the issue w/ below validation
> ```
> assertEquals(asJson(sort(manualColStatsTableDF.drop(colsToDrop: _*), pValidationSortColumns)),
> asJson(sort(pTransposedColStatsDF.drop(colsToDrop: _*), pValidationSortColumns)))
> ```
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)