You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Alexey Kudinkin (Jira)" <ji...@apache.org> on 2022/01/25 22:02:00 UTC
[jira] [Updated] (HUDI-3322) Rollback of Delta Commits performed incorrectly for MOR tables
[ https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Kudinkin updated HUDI-3322:
----------------------------------
Priority: Blocker (was: Major)
> Rollback of Delta Commits performed incorrectly for MOR tables
> --------------------------------------------------------------
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Alexey Kudinkin
> Priority: Blocker
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause of the problem is actually a Rollback of Delta Commits that is performed incorrectly for MOR tables. Consider the case below (we will continue to rely on test of {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx------ 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56 .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57 .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08 .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56 .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 007.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 007.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 13:08 20220125130818473.restore.inflight
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:56 archived
> -rw-r--r-- 1 alexey.kudinkin staff 347 Jan 25 12:56 hoodie.properties
> drwxr-xr-x 4 alexey.kudinkin staff 128 Jan 25 12:56 metadata {code}
> Commit "007" metadata:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % cat .hoodie/007.deltacommit
> {
> "partitionToWriteStats" : {
> "2016/03/15" : [ {
> "fileId" : "a685909e-14e8-4adf-b6a2-e0b669e89e61-0",
> "path" : "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471",
> "prevCommit" : "005",
> "numWrites" : 57,
> "numDeletes" : 0,
> "numUpdateWrites" : 57,
> "numInserts" : 0,
> "totalWriteBytes" : 18054,
> "totalWriteErrors" : 0,
> "tempPath" : null,
> "partitionPath" : "2016/03/15",
> "totalLogRecords" : 0,
> "totalLogFilesCompacted" : 0,
> "totalLogSizeCompacted" : 0,
> "totalUpdatedRecordsCompacted" : 0,
> "totalLogBlocks" : 0,
> "totalCorruptLogBlock" : 0,
> "totalRollbackBlocks" : 0,
> "fileSizeInBytes" : 18054,
> "minEventTime" : null,
> "maxEventTime" : null,
> "logVersion" : 2,
> "logOffset" : 0,
> "baseFile" : "a685909e-14e8-4adf-b6a2-e0b669e89e61-0_2-200-323_005.parquet",
> "logFiles" : [ ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398", ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" ]
> } ],
> "2015/03/16" : [ {
> "fileId" : "0ac88bb7-ebd4-4384-94f1-593e6827a868-0",
> "path" : "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472",
> "prevCommit" : "005",
> "numWrites" : 69,
> "numDeletes" : 0,
> "numUpdateWrites" : 69,
> "numInserts" : 0,
> "totalWriteBytes" : 21457,
> "totalWriteErrors" : 0,
> "tempPath" : null,
> "partitionPath" : "2015/03/16",
> "totalLogRecords" : 0,
> "totalLogFilesCompacted" : 0,
> "totalLogSizeCompacted" : 0,
> "totalUpdatedRecordsCompacted" : 0,
> "totalLogBlocks" : 0,
> "totalCorruptLogBlock" : 0,
> "totalRollbackBlocks" : 0,
> "fileSizeInBytes" : 21457,
> "minEventTime" : null,
> "maxEventTime" : null,
> "logVersion" : 2,
> "logOffset" : 0,
> "baseFile" : "0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-200-322_005.parquet",
> "logFiles" : [ ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399", ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472" ]
> } ],
> "2015/03/17" : [ {
> "fileId" : "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0",
> "path" : "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473",
> "prevCommit" : "005",
> "numWrites" : 74,
> "numDeletes" : 0,
> "numUpdateWrites" : 74,
> "numInserts" : 0,
> "totalWriteBytes" : 22889,
> "totalWriteErrors" : 0,
> "tempPath" : null,
> "partitionPath" : "2015/03/17",
> "totalLogRecords" : 0,
> "totalLogFilesCompacted" : 0,
> "totalLogSizeCompacted" : 0,
> "totalUpdatedRecordsCompacted" : 0,
> "totalLogBlocks" : 0,
> "totalCorruptLogBlock" : 0,
> "totalRollbackBlocks" : 0,
> "fileSizeInBytes" : 22889,
> "minEventTime" : null,
> "maxEventTime" : null,
> "logVersion" : 2,
> "logOffset" : 0,
> "baseFile" : "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_0-200-321_005.parquet",
> "logFiles" : [ ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400", ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473" ]
> } ]
> },
> "compacted" : false,
> "extraMetadata" : {
> "schema" : "{\"type\":\"record\",\"name\":\"triprec\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\"},{\"name\":\"_row_key\",\"type\":\"string\"},{\"name\":\"partition_path\",\"type\":\"string\"},{\"name\":\"rider\",\"type\":\"string\"},{\"name\":\"driver\",\"type\":\"string\"},{\"name\":\"begin_lat\",\"type\":\"double\"},{\"name\":\"begin_lon\",\"type\":\"double\"},{\"name\":\"end_lat\",\"type\":\"double\"},{\"name\":\"end_lon\",\"type\":\"double\"},{\"name\":\"distance_in_meters\",\"type\":\"int\"},{\"name\":\"seconds_since_epoch\",\"type\":\"long\"},{\"name\":\"weight\",\"type\":\"float\"},{\"name\":\"nation\",\"type\":\"bytes\"},{\"name\":\"current_date\",\"type\":{\"type\":\"int\",\"logicalType\":\"date\"}},{\"name\":\"current_ts\",\"type\":\"long\"},{\"name\":\"height\",\"type\":{\"type\":\"fixed\",\"name\":\"abc\",\"size\":5,\"logicalType\":\"decimal\",\"precision\":10,\"scale\":6}},{\"name\":\"city_to_state\",\"type\":{\"type\":\"map\",\"values\":\"string\"}},{\"name\":\"fare\",\"type\":{\"type\":\"record\",\"name\":\"fare\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}]}},{\"name\":\"tip_history\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"tip_history\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}],\"default\":null},\"default\":[]},\"default\":[]},{\"name\":\"_hoodie_is_deleted\",\"type\":\"boolean\",\"default\":false}]}"
> },
> "operationType" : "UPSERT",
> "totalLogFilesCompacted" : 0,
> "fileIdAndRelativePaths" : {
> "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0" : "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473",
> "0ac88bb7-ebd4-4384-94f1-593e6827a868-0" : "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472",
> "a685909e-14e8-4adf-b6a2-e0b669e89e61-0" : "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471"
> },
> "totalLogRecordsCompacted" : 0,
> "totalCompactedRecordsUpdated" : 0,
> "totalRecordsDeleted" : 0,
> "totalLogFilesSize" : 0,
> "totalScanTime" : 0,
> "totalCreateTime" : 0,
> "totalUpsertTime" : 139,
> "minAndMaxEventTime" : {
> "Optional.empty" : {
> "val" : null,
> "present" : false
> }
> },
> "writePartitionPaths" : [ "2016/03/15", "2015/03/16", "2015/03/17" ]
> }% {code}
>
> From the metadata it's not hard to see that only 3 files have changed as part of "007" commit:
> {code:java}
> "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473"
> "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472"
> "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" {code}
> However, rollback of the "007" instant produces following plan:
> {code:java}
> {
> "instantToRollback": {
> "commitTime": "007",
> "action": "deltacommit"
> },
> "RollbackRequests": [
> {
> "partitionPath": "2015/03/16",
> "fileId": "",
> "latestBaseInstant": "",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {}
> },
> {
> "partitionPath": "2015/03/16",
> "fileId": "0ac88bb7-ebd4-4384-94f1-593e6827a868-0",
> "latestBaseInstant": "005",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399": 21464,
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472": 21457
> }
> },
> {
> "partitionPath": "2015/03/17",
> "fileId": "",
> "latestBaseInstant": "",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {}
> },
> {
> "partitionPath": "2015/03/17",
> "fileId": "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0",
> "latestBaseInstant": "005",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473": 22889,
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400": 22881
> }
> },
> {
> "partitionPath": "2016/03/15",
> "fileId": "",
> "latestBaseInstant": "",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {}
> },
> {
> "partitionPath": "2016/03/15",
> "fileId": "a685909e-14e8-4adf-b6a2-e0b669e89e61-0",
> "latestBaseInstant": "005",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398": 18022,
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471": 18054
> }
> }
> ],
> "version": 1
> } {code}
> Which contains files that have not been changed in the "007" instant at all.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)