You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "coffee34 (via GitHub)" <gi...@apache.org> on 2023/03/08 13:31:21 UTC
[GitHub] [hudi] coffee34 opened a new issue, #8132: [SUPPORT] data loss in new base file after compaction
coffee34 opened a new issue, #8132:
URL: https://github.com/apache/hudi/issues/8132
**Describe the problem you faced**
I encountered data loss in hudi mor table. After compaction, a base file became smaller and lost data. The issue occurred on 2023-01-10 and I am only able to access archived commits.(The parquet and log files for those instants have already been deleted.) I have uploaded all the commits. Here is the timeline of the basefile: **8fb29db6-81da-4455-a08d-ba3e7ee36856-0**, it shows that its size was 116M until L23. However, it became 66M after compaction and lost 890k records.
lineNum | commitTime | actionType | actionState | Plan
---------|---------------|---------------|---------------|---------------------------------------
L16 | 20230110133240 | compaction | REQUESTED | {"baseInstantTime": "20230110130744", "deltaFilePaths": [".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110130744.log.1_53-7076606-255800368"], "dataFilePath": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0_24-7076286-255788001_20230110130744.parquet", "fileId": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0", "partitionPath": "daas_date=2022", "metrics": {"TOTAL_LOG_FILES": 1.0, "TOTAL_IO_READ_MB": 116.0, "TOTAL_LOG_FILES_SIZE": 24243.0, "TOTAL_IO_WRITE_MB": 116.0, "TOTAL_IO_MB": 232.0}, "bootstrapFilePath": null}
L17 | 20230110133240 | compaction | INFLIGHT | {"baseInstantTime": "20230110130744", "deltaFilePaths": [".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110130744.log.1_53-7076606-255800368"], "dataFilePath": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0_24-7076286-255788001_20230110130744.parquet", "fileId": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0", "partitionPath": "daas_date=2022", "metrics": {"TOTAL_LOG_FILES": 1.0, "TOTAL_IO_READ_MB": 116.0, "TOTAL_LOG_FILES_SIZE": 24243.0, "TOTAL_IO_WRITE_MB": 116.0, "TOTAL_IO_MB": 232.0}
L18 | 20230110133240 | commit | COMPLETED | {\"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"path\": \"daas_date=2022/8fb29db6-81da-4455-a08d-ba3e7ee36856-0_27-7076664-255801644_20230110133240.parquet\", \"prevCommit\": \"20230110130744\", \"numWrites\": 2060123, \"numDeletes\": 0, \"numUpdateWrites\": 99, \"totalWriteBytes\": 122155577, \"totalWriteErrors\": 0, \"partitionPath\": \"daas_date=2022\", \"totalLogRecords\": 99, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 99, \"numInserts\": 0, \"totalLogBlocks\": 1, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 122155577}
L19 | 20230110135603 | commit | REQUESTED |
L20 | 20230110135603 | commit | INFLIGHT | {\"fileId\": \"3c82ba35-b701-4f34-882a-167146036ab3-0\", \"path\": null, \"prevCommit\": \"20230110133240\", \"numWrites\": 0, \"numDeletes\": 0, \"numUpdateWrites\": 108, \"totalWriteBytes\": 0, \"totalWriteErrors\": 0, \"partitionPath\": null, \"totalLogRecords\": 0, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 0, \"numInserts\": 0, \"totalLogBlocks\": 0, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 0}
L21 | 20230110135603 | commit | COMPLETED | {\"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"path\": \"daas_date=2022/.8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110133240.log.1_53-7077163-255815146\", \"prevCommit\": \"20230110133240\", \"numWrites\": 106, \"numDeletes\": 0, \"numUpdateWrites\": 106, \"totalWriteBytes\": 25777, \"totalWriteErrors\": 0, \"partitionPath\": \"daas_date=2022\", \"totalLogRecords\": 0, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 0, \"numInserts\": 0, \"totalLogBlocks\": 0, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 25777}
L22 | 20230110135817 | compaction | REQUESTED | {\"baseInstantTime\": \"20230110133240\", \"deltaFilePaths\": [\".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110133240.log.1_53-7077163-255815146\"], \"dataFilePath\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0_27-7076664-255801644_20230110133240.parquet\", \"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"partitionPath\": \"daas_date=2022\", \"metrics\": {\"TOTAL_LOG_FILES\": 1.0, \"TOTAL_IO_READ_MB\": 116.0, \"TOTAL_LOG_FILES_SIZE\": 25777.0, \"TOTAL_IO_WRITE_MB\": 116.0, \"TOTAL_IO_MB\": 232.0}, \"bootstrapFilePath\": null}
L23 | 20230110135817 | compaction | INFLIGHT | {\"baseInstantTime\": \"20230110133240\", \"deltaFilePaths\": [\".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110133240.log.1_53-7077163-255815146\"], \"dataFilePath\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0_27-7076664-255801644_20230110133240.parquet\", \"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"partitionPath\": \"daas_date=2022\", \"metrics\": {\"TOTAL_LOG_FILES\": 1.0, \"TOTAL_IO_READ_MB\": 116.0, \"TOTAL_LOG_FILES_SIZE\": 25777.0, \"TOTAL_IO_WRITE_MB\": 116.0, \"TOTAL_IO_MB\": 232.0}, \"bootstrapFilePath\": null}
L24 | 20230110135817 | commit | COMPLETED | {\"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"path\": \"daas_date=2022/8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20-7077198-255819567_20230110135817.parquet\", \"prevCommit\": \"20230110133240\", \"numWrites\": 1169659, \"numDeletes\": 0, \"numUpdateWrites\": 54, \"totalWriteBytes\": 69228459, \"totalWriteErrors\": 0, \"partitionPath\": \"daas_date=2022\", \"totalLogRecords\": 106, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 106, \"numInserts\": 52, \"totalLogBlocks\": 1, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 69228459}
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.7.0
* Spark version :2.4.4
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional [context**]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #8132: [SUPPORT] data loss in new base file after compaction
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #8132:
URL: https://github.com/apache/hudi/issues/8132#issuecomment-1473201894
got it. thanks. since this is w/ 0.7.0, I am not sure if we had fixed anything on this already. is it feasible to upgrade to 012.2 or 0.13.0 ? you will definitely get all new features, perf etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8132: [SUPPORT] data loss in new base file after compaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8132:
URL: https://github.com/apache/hudi/issues/8132#issuecomment-1461380796
Yeah, we found another data loss issue: https://github.com/apache/hudi/pull/8079, let's wait for the release for 0.13.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #8132: [SUPPORT] data loss in new base file after compaction
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #8132:
URL: https://github.com/apache/hudi/issues/8132#issuecomment-1465087986
I can't seem to find any reason why this could happen. but don't think 8079 is the issue. that would surface differently.
anyways, 0.7.0 is very old. We have come a long way from 0.7.0. We have fixed issues w/ compaction and spark cache invalidation
For eg https://github.com/apache/hudi/pull/4753
https://github.com/apache/hudi/pull/4856
but I could not exactly say if these are the issues.
But from the timeline provided, I could not reason about why parquet size might reduce. infact number of records reduced by 50% from previous version, even though log file modified just 100 ish records :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8132: [SUPPORT] data loss in new base file after compaction
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8132:
URL: https://github.com/apache/hudi/issues/8132#issuecomment-1491930079
@coffee34 Have you tried upgrading the Hudi version, Are you still facing this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] coffee34 commented on issue #8132: [SUPPORT] data loss in new base file after compaction
Posted by "coffee34 (via GitHub)" <gi...@apache.org>.
coffee34 commented on issue #8132:
URL: https://github.com/apache/hudi/issues/8132#issuecomment-1460179532
Based on the information provided, it appears that there were 106 updates at L21, but at L24, there were only 54 updates and 52 inserts. The related code for this behavior can be found at https://github.com/apache/hudi/blob/162dc18fc6a1e1d0db420a4735bc8c5a0ba7cf12/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java#L265.
It seems that the developers are already aware that updates can sometimes be turned into inserts. Can you please provide more information on the specific conditions that could cause this behavior to occur?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #8132: [SUPPORT] data loss in new base file after compaction
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #8132:
URL: https://github.com/apache/hudi/issues/8132#issuecomment-1527027664
hey @coffee34 : can you help us w/ any more info on this end. we are taking a serious look into all data consistency issues. So, interested in getting to the bottom of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] coffee34 commented on issue #8132: [SUPPORT] data loss in new base file after compaction
Posted by "coffee34 (via GitHub)" <gi...@apache.org>.
coffee34 commented on issue #8132:
URL: https://github.com/apache/hudi/issues/8132#issuecomment-1471301250
Thanks for reply.
Currently, we have only one writer running, and it has been running without any errors for over half a year. However, I have set up a monitoring system to detect if this issue occurs again, and I am trying to find a way to reproduce it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #8132: [SUPPORT] data loss in new base file after compaction
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #8132:
URL: https://github.com/apache/hudi/issues/8132#issuecomment-1505022212
@coffee34 Can you please try upgrading and if you still see data loss, can you share the write configurations and timeline?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #8132: [SUPPORT] data loss in new base file after compaction
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #8132:
URL: https://github.com/apache/hudi/issues/8132#issuecomment-1465088037
Do you know if there could be any unintentional multi-writer interplay?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #8132: [SUPPORT] data loss in new base file after compaction
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #8132:
URL: https://github.com/apache/hudi/issues/8132#issuecomment-1465087152
I am not sure if this was related to https://github.com/apache/hudi/pull/8079
I am trying to analyze all details. will update if I have any findings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #8132: [SUPPORT] data loss in new base file after compaction
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #8132:
URL: https://github.com/apache/hudi/issues/8132#issuecomment-1528118945
hey @coffee34 : The code that you pointed out where inserts converted to updates is not applicable to bloom and simple index. Those are meant for few other index types. But for these two indexes, inserts will always go into a new base file and only update can go into log files.
From the info provided, we could not get any leads and we don't see any other data loss reports from the community. We do have some reports around global index, but those will surface differently.
What you are reporting is completely diff. Can you provide us w/ a full backup of ".hoodie" and write configs. but as of now, we can't think of any reason why such huge data loss can occur.
unless you changed the record key / partition path config for instance. Do you happen to know any changes to your pipelines around the time where you see data loss.
like any config updates. or anything of those sort.
If you have a back up of the file slice in question, we can try to copy them locally(on your end) and trigger compaction and debug whats going on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org