You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Zeyu Wang (Jira)" <ji...@apache.org> on 2024/04/19 08:41:00 UTC

[jira] [Created] (HUDI-7642) Compact MOR tables with operation fields cause data errors

Zeyu Wang created HUDI-7642:
-------------------------------

             Summary: Compact MOR tables with operation fields cause data errors
                 Key: HUDI-7642
                 URL: https://issues.apache.org/jira/browse/HUDI-7642
             Project: Apache Hudi
          Issue Type: Bug
            Reporter: Zeyu Wang


When we compact an MOR table who with _hoodie_operation field, the hoodiekey tagged with operation "-D" was not correctly removed.

Refer to previous discussions (https://github.com/apache/hudi/pull/8721#issuecomment-1736629662) we should keep flink engine for the delete record, And also repair the spark in the https://github.com/apache/hudi/pull/10219 engine problems when reading data, should repair caused by compact problem now. Because of the 'compact' directly using theĀ  HoodieMergedLogRecordScanner in the common module, I think we have to add some optional configuration to control whether or not the HoodieMergedLogRecordScanner directly delete the key that taged with "-D" operation



--
This message was sent by Atlassian Jira
(v8.20.10#820010)