You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/17 08:56:32 UTC
[GitHub] [hudi] hj2016 opened a new pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
hj2016 opened a new pull request #4015:
URL: https://github.com/apache/hudi/pull/4015
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Fix Mor reads the log file and skips the complete block as a bad block, resulting in data loss
jira issues https://issues.apache.org/jira/browse/HUDI-2780
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4015:
URL: https://github.com/apache/hudi/pull/4015#issuecomment-1008360786
@alexeykudinkin : Is there any follow up on this patch. Let's sync up sometime to see if we know what's the right fix and if we can squeeze it in for 0.10.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4015:
URL: https://github.com/apache/hudi/pull/4015#discussion_r770865659
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
##########
@@ -291,7 +292,7 @@ private boolean isBlockCorrupt(int blocksize) throws IOException {
// This seek is required because contract of seek() is different for naked DFSInputStream vs BufferedFSInputStream
// release-3.1.0-RC1/DFSInputStream.java#L1455
// release-3.1.0-RC1/BufferedFSInputStream.java#L73
- inputStream.seek(currentPos);
+ inputStream.seek(blockStartPos);
Review comment:
A few things here:
1. `isBlockCorrupt` (should be `Corrupted` actually) is method that checks the state of the stream, and as such should not have any side-effects not expected from it -- it should not rewind stream to any other position other than it was when this method have been invoked. If we want to rewind the stream to some other position we should do this outside of this method -- we can do that right where this method is invoked.
2. I read through the ticket, and i think i've got the hold of the issue you're facing but i don't think i understand your solution. Can you please add comments to the code to explain the semantic you're trying to set here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4015:
URL: https://github.com/apache/hudi/pull/4015#issuecomment-971370531
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "375927ade5b4b327e44ebc227fb57e64de524fcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "375927ade5b4b327e44ebc227fb57e64de524fcc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 375927ade5b4b327e44ebc227fb57e64de524fcc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4015:
URL: https://github.com/apache/hudi/pull/4015#issuecomment-971418416
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "375927ade5b4b327e44ebc227fb57e64de524fcc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3426",
"triggerID" : "375927ade5b4b327e44ebc227fb57e64de524fcc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 375927ade5b4b327e44ebc227fb57e64de524fcc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3426)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #4015:
URL: https://github.com/apache/hudi/pull/4015#issuecomment-994232638
cc @alexeykudinkin can you take a pass at this please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4015:
URL: https://github.com/apache/hudi/pull/4015#issuecomment-971370531
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "375927ade5b4b327e44ebc227fb57e64de524fcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "375927ade5b4b327e44ebc227fb57e64de524fcc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 375927ade5b4b327e44ebc227fb57e64de524fcc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4015:
URL: https://github.com/apache/hudi/pull/4015#issuecomment-971372854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "375927ade5b4b327e44ebc227fb57e64de524fcc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3426",
"triggerID" : "375927ade5b4b327e44ebc227fb57e64de524fcc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 375927ade5b4b327e44ebc227fb57e64de524fcc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3426)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4015:
URL: https://github.com/apache/hudi/pull/4015#issuecomment-1050513786
@hj2016 : can you please help clarify things. we want to see if there is a real bug, we want to fix it before 0.11.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4015:
URL: https://github.com/apache/hudi/pull/4015#discussion_r770865659
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
##########
@@ -291,7 +292,7 @@ private boolean isBlockCorrupt(int blocksize) throws IOException {
// This seek is required because contract of seek() is different for naked DFSInputStream vs BufferedFSInputStream
// release-3.1.0-RC1/DFSInputStream.java#L1455
// release-3.1.0-RC1/BufferedFSInputStream.java#L73
- inputStream.seek(currentPos);
+ inputStream.seek(blockStartPos);
Review comment:
A few things here:
1. `isBlockCorrupt` (should be `Corrupted` actually) is method that checks the state of the stream, and as such should not have any side-effects not expected from it -- it should not rewind stream to any other position other than the one it was in when this method have been invoked. If we want to rewind the stream to some other position we should do this outside of this method -- we can do that right where this method is invoked.
2. I read through the ticket, and i think i've got the hold of the issue you're facing but i don't think i understand your solution. Can you please add comments to the code to explain the semantic you're trying to set here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4015:
URL: https://github.com/apache/hudi/pull/4015#issuecomment-998303637
I will let @alexeykudinkin take care of this patch going forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4015:
URL: https://github.com/apache/hudi/pull/4015#issuecomment-971372854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "375927ade5b4b327e44ebc227fb57e64de524fcc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3426",
"triggerID" : "375927ade5b4b327e44ebc227fb57e64de524fcc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 375927ade5b4b327e44ebc227fb57e64de524fcc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3426)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org