You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Alexey Kudinkin (Jira)" <ji...@apache.org> on 2022/01/25 22:40:00 UTC
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[ https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Kudinkin updated HUDI-3322:
----------------------------------
Description:
Diving deeper into the issue of HUDI-3279, i've realized that the root-cause of the problem is actually a Rollback Plan for Delta Commits is composed incorrectly for MOR tables. Consider the case below (we will continue to rely on test of {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
Hoodie Timeline:
{code:java}
alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
total 400
drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
drwx------ 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56 .001.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .001.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .002.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .002.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .003.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .003.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .004.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .004.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .005.compaction.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57 .005.compaction.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .006.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .006.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .007.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .007.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08 .20220125130818473.restore.inflight.crc
drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
-rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56 .hoodie.properties.crc
drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
-rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56 001.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 001.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56 002.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 002.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 003.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 003.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 004.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 004.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 005.compaction.inflight
-rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57 005.compaction.requested
-rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 006.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 006.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 007.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 007.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 13:08 20220125130818473.restore.inflight
drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:56 archived
-rw-r--r-- 1 alexey.kudinkin staff 347 Jan 25 12:56 hoodie.properties
drwxr-xr-x 4 alexey.kudinkin staff 128 Jan 25 12:56 metadata {code}
Commit "007" metadata:
{code:java}
alexey.kudinkin@alexeys-mbp junit5494198038159268501 % cat .hoodie/007.deltacommit
{
"partitionToWriteStats" : {
"2016/03/15" : [ {
"fileId" : "a685909e-14e8-4adf-b6a2-e0b669e89e61-0",
"path" : "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471",
"prevCommit" : "005",
"numWrites" : 57,
"numDeletes" : 0,
"numUpdateWrites" : 57,
"numInserts" : 0,
"totalWriteBytes" : 18054,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "2016/03/15",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 18054,
"minEventTime" : null,
"maxEventTime" : null,
"logVersion" : 2,
"logOffset" : 0,
"baseFile" : "a685909e-14e8-4adf-b6a2-e0b669e89e61-0_2-200-323_005.parquet",
"logFiles" : [ ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398", ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" ]
} ],
"2015/03/16" : [ {
"fileId" : "0ac88bb7-ebd4-4384-94f1-593e6827a868-0",
"path" : "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472",
"prevCommit" : "005",
"numWrites" : 69,
"numDeletes" : 0,
"numUpdateWrites" : 69,
"numInserts" : 0,
"totalWriteBytes" : 21457,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "2015/03/16",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 21457,
"minEventTime" : null,
"maxEventTime" : null,
"logVersion" : 2,
"logOffset" : 0,
"baseFile" : "0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-200-322_005.parquet",
"logFiles" : [ ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399", ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472" ]
} ],
"2015/03/17" : [ {
"fileId" : "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0",
"path" : "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473",
"prevCommit" : "005",
"numWrites" : 74,
"numDeletes" : 0,
"numUpdateWrites" : 74,
"numInserts" : 0,
"totalWriteBytes" : 22889,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "2015/03/17",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 22889,
"minEventTime" : null,
"maxEventTime" : null,
"logVersion" : 2,
"logOffset" : 0,
"baseFile" : "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_0-200-321_005.parquet",
"logFiles" : [ ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400", ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473" ]
} ]
},
"compacted" : false,
"extraMetadata" : {
"schema" : "{\"type\":\"record\",\"name\":\"triprec\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\"},{\"name\":\"_row_key\",\"type\":\"string\"},{\"name\":\"partition_path\",\"type\":\"string\"},{\"name\":\"rider\",\"type\":\"string\"},{\"name\":\"driver\",\"type\":\"string\"},{\"name\":\"begin_lat\",\"type\":\"double\"},{\"name\":\"begin_lon\",\"type\":\"double\"},{\"name\":\"end_lat\",\"type\":\"double\"},{\"name\":\"end_lon\",\"type\":\"double\"},{\"name\":\"distance_in_meters\",\"type\":\"int\"},{\"name\":\"seconds_since_epoch\",\"type\":\"long\"},{\"name\":\"weight\",\"type\":\"float\"},{\"name\":\"nation\",\"type\":\"bytes\"},{\"name\":\"current_date\",\"type\":{\"type\":\"int\",\"logicalType\":\"date\"}},{\"name\":\"current_ts\",\"type\":\"long\"},{\"name\":\"height\",\"type\":{\"type\":\"fixed\",\"name\":\"abc\",\"size\":5,\"logicalType\":\"decimal\",\"precision\":10,\"scale\":6}},{\"name\":\"city_to_state\",\"type\":{\"type\":\"map\",\"values\":\"string\"}},{\"name\":\"fare\",\"type\":{\"type\":\"record\",\"name\":\"fare\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}]}},{\"name\":\"tip_history\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"tip_history\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}],\"default\":null},\"default\":[]},\"default\":[]},{\"name\":\"_hoodie_is_deleted\",\"type\":\"boolean\",\"default\":false}]}"
},
"operationType" : "UPSERT",
"totalLogFilesCompacted" : 0,
"fileIdAndRelativePaths" : {
"52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0" : "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473",
"0ac88bb7-ebd4-4384-94f1-593e6827a868-0" : "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472",
"a685909e-14e8-4adf-b6a2-e0b669e89e61-0" : "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471"
},
"totalLogRecordsCompacted" : 0,
"totalCompactedRecordsUpdated" : 0,
"totalRecordsDeleted" : 0,
"totalLogFilesSize" : 0,
"totalScanTime" : 0,
"totalCreateTime" : 0,
"totalUpsertTime" : 139,
"minAndMaxEventTime" : {
"Optional.empty" : {
"val" : null,
"present" : false
}
},
"writePartitionPaths" : [ "2016/03/15", "2015/03/16", "2015/03/17" ]
}% {code}
From the metadata it's not hard to see that only 3 files have changed as part of "007" commit:
{code:java}
"2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473"
"2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472"
"2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" {code}
However, rollback of the "007" instant produces following plan:
{code:java}
{
"instantToRollback": {
"commitTime": "007",
"action": "deltacommit"
},
"RollbackRequests": [
{
"partitionPath": "2015/03/16",
"fileId": "",
"latestBaseInstant": "",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {}
},
{
"partitionPath": "2015/03/16",
"fileId": "0ac88bb7-ebd4-4384-94f1-593e6827a868-0",
"latestBaseInstant": "005",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399": 21464,
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472": 21457
}
},
{
"partitionPath": "2015/03/17",
"fileId": "",
"latestBaseInstant": "",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {}
},
{
"partitionPath": "2015/03/17",
"fileId": "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0",
"latestBaseInstant": "005",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473": 22889,
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400": 22881
}
},
{
"partitionPath": "2016/03/15",
"fileId": "",
"latestBaseInstant": "",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {}
},
{
"partitionPath": "2016/03/15",
"fileId": "a685909e-14e8-4adf-b6a2-e0b669e89e61-0",
"latestBaseInstant": "005",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398": 18022,
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471": 18054
}
}
],
"version": 1
} {code}
Which contains files that have not been changed in the "007" instant at all.
was:
Diving deeper into the issue of HUDI-3279, i've realized that the root-cause of the problem is actually a Rollback of Delta Commits that is performed incorrectly for MOR tables. Consider the case below (we will continue to rely on test of {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
Hoodie Timeline:
{code:java}
alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
total 400
drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
drwx------ 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56 .001.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .001.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .002.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .002.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .003.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .003.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .004.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .004.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .005.compaction.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57 .005.compaction.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .006.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .006.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .007.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .007.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08 .20220125130818473.restore.inflight.crc
drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
-rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56 .hoodie.properties.crc
drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
-rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56 001.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 001.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56 002.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 002.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 003.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 003.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 004.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 004.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 005.compaction.inflight
-rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57 005.compaction.requested
-rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 006.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 006.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 007.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 007.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 13:08 20220125130818473.restore.inflight
drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:56 archived
-rw-r--r-- 1 alexey.kudinkin staff 347 Jan 25 12:56 hoodie.properties
drwxr-xr-x 4 alexey.kudinkin staff 128 Jan 25 12:56 metadata {code}
Commit "007" metadata:
{code:java}
alexey.kudinkin@alexeys-mbp junit5494198038159268501 % cat .hoodie/007.deltacommit
{
"partitionToWriteStats" : {
"2016/03/15" : [ {
"fileId" : "a685909e-14e8-4adf-b6a2-e0b669e89e61-0",
"path" : "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471",
"prevCommit" : "005",
"numWrites" : 57,
"numDeletes" : 0,
"numUpdateWrites" : 57,
"numInserts" : 0,
"totalWriteBytes" : 18054,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "2016/03/15",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 18054,
"minEventTime" : null,
"maxEventTime" : null,
"logVersion" : 2,
"logOffset" : 0,
"baseFile" : "a685909e-14e8-4adf-b6a2-e0b669e89e61-0_2-200-323_005.parquet",
"logFiles" : [ ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398", ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" ]
} ],
"2015/03/16" : [ {
"fileId" : "0ac88bb7-ebd4-4384-94f1-593e6827a868-0",
"path" : "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472",
"prevCommit" : "005",
"numWrites" : 69,
"numDeletes" : 0,
"numUpdateWrites" : 69,
"numInserts" : 0,
"totalWriteBytes" : 21457,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "2015/03/16",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 21457,
"minEventTime" : null,
"maxEventTime" : null,
"logVersion" : 2,
"logOffset" : 0,
"baseFile" : "0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-200-322_005.parquet",
"logFiles" : [ ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399", ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472" ]
} ],
"2015/03/17" : [ {
"fileId" : "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0",
"path" : "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473",
"prevCommit" : "005",
"numWrites" : 74,
"numDeletes" : 0,
"numUpdateWrites" : 74,
"numInserts" : 0,
"totalWriteBytes" : 22889,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "2015/03/17",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 22889,
"minEventTime" : null,
"maxEventTime" : null,
"logVersion" : 2,
"logOffset" : 0,
"baseFile" : "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_0-200-321_005.parquet",
"logFiles" : [ ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400", ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473" ]
} ]
},
"compacted" : false,
"extraMetadata" : {
"schema" : "{\"type\":\"record\",\"name\":\"triprec\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\"},{\"name\":\"_row_key\",\"type\":\"string\"},{\"name\":\"partition_path\",\"type\":\"string\"},{\"name\":\"rider\",\"type\":\"string\"},{\"name\":\"driver\",\"type\":\"string\"},{\"name\":\"begin_lat\",\"type\":\"double\"},{\"name\":\"begin_lon\",\"type\":\"double\"},{\"name\":\"end_lat\",\"type\":\"double\"},{\"name\":\"end_lon\",\"type\":\"double\"},{\"name\":\"distance_in_meters\",\"type\":\"int\"},{\"name\":\"seconds_since_epoch\",\"type\":\"long\"},{\"name\":\"weight\",\"type\":\"float\"},{\"name\":\"nation\",\"type\":\"bytes\"},{\"name\":\"current_date\",\"type\":{\"type\":\"int\",\"logicalType\":\"date\"}},{\"name\":\"current_ts\",\"type\":\"long\"},{\"name\":\"height\",\"type\":{\"type\":\"fixed\",\"name\":\"abc\",\"size\":5,\"logicalType\":\"decimal\",\"precision\":10,\"scale\":6}},{\"name\":\"city_to_state\",\"type\":{\"type\":\"map\",\"values\":\"string\"}},{\"name\":\"fare\",\"type\":{\"type\":\"record\",\"name\":\"fare\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}]}},{\"name\":\"tip_history\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"tip_history\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}],\"default\":null},\"default\":[]},\"default\":[]},{\"name\":\"_hoodie_is_deleted\",\"type\":\"boolean\",\"default\":false}]}"
},
"operationType" : "UPSERT",
"totalLogFilesCompacted" : 0,
"fileIdAndRelativePaths" : {
"52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0" : "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473",
"0ac88bb7-ebd4-4384-94f1-593e6827a868-0" : "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472",
"a685909e-14e8-4adf-b6a2-e0b669e89e61-0" : "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471"
},
"totalLogRecordsCompacted" : 0,
"totalCompactedRecordsUpdated" : 0,
"totalRecordsDeleted" : 0,
"totalLogFilesSize" : 0,
"totalScanTime" : 0,
"totalCreateTime" : 0,
"totalUpsertTime" : 139,
"minAndMaxEventTime" : {
"Optional.empty" : {
"val" : null,
"present" : false
}
},
"writePartitionPaths" : [ "2016/03/15", "2015/03/16", "2015/03/17" ]
}% {code}
From the metadata it's not hard to see that only 3 files have changed as part of "007" commit:
{code:java}
"2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473"
"2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472"
"2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" {code}
However, rollback of the "007" instant produces following plan:
{code:java}
{
"instantToRollback": {
"commitTime": "007",
"action": "deltacommit"
},
"RollbackRequests": [
{
"partitionPath": "2015/03/16",
"fileId": "",
"latestBaseInstant": "",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {}
},
{
"partitionPath": "2015/03/16",
"fileId": "0ac88bb7-ebd4-4384-94f1-593e6827a868-0",
"latestBaseInstant": "005",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399": 21464,
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472": 21457
}
},
{
"partitionPath": "2015/03/17",
"fileId": "",
"latestBaseInstant": "",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {}
},
{
"partitionPath": "2015/03/17",
"fileId": "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0",
"latestBaseInstant": "005",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473": 22889,
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400": 22881
}
},
{
"partitionPath": "2016/03/15",
"fileId": "",
"latestBaseInstant": "",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {}
},
{
"partitionPath": "2016/03/15",
"fileId": "a685909e-14e8-4adf-b6a2-e0b669e89e61-0",
"latestBaseInstant": "005",
"filesToBeDeleted": [],
"logBlocksToBeDeleted": {
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398": 18022,
"file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471": 18054
}
}
],
"version": 1
} {code}
Which contains files that have not been changed in the "007" instant at all.
> Rollback Plan for Delta Commits constructed incorrectly
> -------------------------------------------------------
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Alexey Kudinkin
> Priority: Blocker
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause of the problem is actually a Rollback Plan for Delta Commits is composed incorrectly for MOR tables. Consider the case below (we will continue to rely on test of {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx------ 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56 .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57 .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08 .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56 .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 007.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 007.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 13:08 20220125130818473.restore.inflight
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:56 archived
> -rw-r--r-- 1 alexey.kudinkin staff 347 Jan 25 12:56 hoodie.properties
> drwxr-xr-x 4 alexey.kudinkin staff 128 Jan 25 12:56 metadata {code}
> Commit "007" metadata:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % cat .hoodie/007.deltacommit
> {
> "partitionToWriteStats" : {
> "2016/03/15" : [ {
> "fileId" : "a685909e-14e8-4adf-b6a2-e0b669e89e61-0",
> "path" : "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471",
> "prevCommit" : "005",
> "numWrites" : 57,
> "numDeletes" : 0,
> "numUpdateWrites" : 57,
> "numInserts" : 0,
> "totalWriteBytes" : 18054,
> "totalWriteErrors" : 0,
> "tempPath" : null,
> "partitionPath" : "2016/03/15",
> "totalLogRecords" : 0,
> "totalLogFilesCompacted" : 0,
> "totalLogSizeCompacted" : 0,
> "totalUpdatedRecordsCompacted" : 0,
> "totalLogBlocks" : 0,
> "totalCorruptLogBlock" : 0,
> "totalRollbackBlocks" : 0,
> "fileSizeInBytes" : 18054,
> "minEventTime" : null,
> "maxEventTime" : null,
> "logVersion" : 2,
> "logOffset" : 0,
> "baseFile" : "a685909e-14e8-4adf-b6a2-e0b669e89e61-0_2-200-323_005.parquet",
> "logFiles" : [ ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398", ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" ]
> } ],
> "2015/03/16" : [ {
> "fileId" : "0ac88bb7-ebd4-4384-94f1-593e6827a868-0",
> "path" : "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472",
> "prevCommit" : "005",
> "numWrites" : 69,
> "numDeletes" : 0,
> "numUpdateWrites" : 69,
> "numInserts" : 0,
> "totalWriteBytes" : 21457,
> "totalWriteErrors" : 0,
> "tempPath" : null,
> "partitionPath" : "2015/03/16",
> "totalLogRecords" : 0,
> "totalLogFilesCompacted" : 0,
> "totalLogSizeCompacted" : 0,
> "totalUpdatedRecordsCompacted" : 0,
> "totalLogBlocks" : 0,
> "totalCorruptLogBlock" : 0,
> "totalRollbackBlocks" : 0,
> "fileSizeInBytes" : 21457,
> "minEventTime" : null,
> "maxEventTime" : null,
> "logVersion" : 2,
> "logOffset" : 0,
> "baseFile" : "0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-200-322_005.parquet",
> "logFiles" : [ ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399", ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472" ]
> } ],
> "2015/03/17" : [ {
> "fileId" : "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0",
> "path" : "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473",
> "prevCommit" : "005",
> "numWrites" : 74,
> "numDeletes" : 0,
> "numUpdateWrites" : 74,
> "numInserts" : 0,
> "totalWriteBytes" : 22889,
> "totalWriteErrors" : 0,
> "tempPath" : null,
> "partitionPath" : "2015/03/17",
> "totalLogRecords" : 0,
> "totalLogFilesCompacted" : 0,
> "totalLogSizeCompacted" : 0,
> "totalUpdatedRecordsCompacted" : 0,
> "totalLogBlocks" : 0,
> "totalCorruptLogBlock" : 0,
> "totalRollbackBlocks" : 0,
> "fileSizeInBytes" : 22889,
> "minEventTime" : null,
> "maxEventTime" : null,
> "logVersion" : 2,
> "logOffset" : 0,
> "baseFile" : "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_0-200-321_005.parquet",
> "logFiles" : [ ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400", ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473" ]
> } ]
> },
> "compacted" : false,
> "extraMetadata" : {
> "schema" : "{\"type\":\"record\",\"name\":\"triprec\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\"},{\"name\":\"_row_key\",\"type\":\"string\"},{\"name\":\"partition_path\",\"type\":\"string\"},{\"name\":\"rider\",\"type\":\"string\"},{\"name\":\"driver\",\"type\":\"string\"},{\"name\":\"begin_lat\",\"type\":\"double\"},{\"name\":\"begin_lon\",\"type\":\"double\"},{\"name\":\"end_lat\",\"type\":\"double\"},{\"name\":\"end_lon\",\"type\":\"double\"},{\"name\":\"distance_in_meters\",\"type\":\"int\"},{\"name\":\"seconds_since_epoch\",\"type\":\"long\"},{\"name\":\"weight\",\"type\":\"float\"},{\"name\":\"nation\",\"type\":\"bytes\"},{\"name\":\"current_date\",\"type\":{\"type\":\"int\",\"logicalType\":\"date\"}},{\"name\":\"current_ts\",\"type\":\"long\"},{\"name\":\"height\",\"type\":{\"type\":\"fixed\",\"name\":\"abc\",\"size\":5,\"logicalType\":\"decimal\",\"precision\":10,\"scale\":6}},{\"name\":\"city_to_state\",\"type\":{\"type\":\"map\",\"values\":\"string\"}},{\"name\":\"fare\",\"type\":{\"type\":\"record\",\"name\":\"fare\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}]}},{\"name\":\"tip_history\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"tip_history\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}],\"default\":null},\"default\":[]},\"default\":[]},{\"name\":\"_hoodie_is_deleted\",\"type\":\"boolean\",\"default\":false}]}"
> },
> "operationType" : "UPSERT",
> "totalLogFilesCompacted" : 0,
> "fileIdAndRelativePaths" : {
> "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0" : "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473",
> "0ac88bb7-ebd4-4384-94f1-593e6827a868-0" : "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472",
> "a685909e-14e8-4adf-b6a2-e0b669e89e61-0" : "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471"
> },
> "totalLogRecordsCompacted" : 0,
> "totalCompactedRecordsUpdated" : 0,
> "totalRecordsDeleted" : 0,
> "totalLogFilesSize" : 0,
> "totalScanTime" : 0,
> "totalCreateTime" : 0,
> "totalUpsertTime" : 139,
> "minAndMaxEventTime" : {
> "Optional.empty" : {
> "val" : null,
> "present" : false
> }
> },
> "writePartitionPaths" : [ "2016/03/15", "2015/03/16", "2015/03/17" ]
> }% {code}
>
> From the metadata it's not hard to see that only 3 files have changed as part of "007" commit:
> {code:java}
> "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473"
> "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472"
> "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" {code}
> However, rollback of the "007" instant produces following plan:
> {code:java}
> {
> "instantToRollback": {
> "commitTime": "007",
> "action": "deltacommit"
> },
> "RollbackRequests": [
> {
> "partitionPath": "2015/03/16",
> "fileId": "",
> "latestBaseInstant": "",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {}
> },
> {
> "partitionPath": "2015/03/16",
> "fileId": "0ac88bb7-ebd4-4384-94f1-593e6827a868-0",
> "latestBaseInstant": "005",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399": 21464,
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472": 21457
> }
> },
> {
> "partitionPath": "2015/03/17",
> "fileId": "",
> "latestBaseInstant": "",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {}
> },
> {
> "partitionPath": "2015/03/17",
> "fileId": "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0",
> "latestBaseInstant": "005",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473": 22889,
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400": 22881
> }
> },
> {
> "partitionPath": "2016/03/15",
> "fileId": "",
> "latestBaseInstant": "",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {}
> },
> {
> "partitionPath": "2016/03/15",
> "fileId": "a685909e-14e8-4adf-b6a2-e0b669e89e61-0",
> "latestBaseInstant": "005",
> "filesToBeDeleted": [],
> "logBlocksToBeDeleted": {
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398": 18022,
> "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471": 18054
> }
> }
> ],
> "version": 1
> } {code}
> Which contains files that have not been changed in the "007" instant at all.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)