You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/21 13:06:48 UTC
[GitHub] [hudi] boneanxs opened a new pull request, #7264: [HUDI-5253] Could have duplicates if HoodieRealtimePath has log files
boneanxs opened a new pull request, #7264:
URL: https://github.com/apache/hudi/pull/7264
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1323434724
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177",
"triggerID" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ccbbfbae60ff01e765d89add71ba7382e623c5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154)
* 1b07f271c16acd022d7853df55bb0a838b8b30de Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] Could have duplicates if HoodieRealtimePath has log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1322120277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ccbbfbae60ff01e765d89add71ba7382e623c5b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1324680690
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177",
"triggerID" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13205",
"triggerID" : "1324450660",
"triggerType" : "MANUAL"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13213",
"triggerID" : "1324651032",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1b07f271c16acd022d7853df55bb0a838b8b30de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13205) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13213)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
boneanxs commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1329933428
@danny0405 @xushiyan could you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1330147534
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177",
"triggerID" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13205",
"triggerID" : "1324450660",
"triggerType" : "MANUAL"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13213",
"triggerID" : "1324651032",
"triggerType" : "MANUAL"
}, {
"hash" : "d6167ff53f72e8b630dd794c9d1568026e58fba1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13308",
"triggerID" : "d6167ff53f72e8b630dd794c9d1568026e58fba1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b07f271c16acd022d7853df55bb0a838b8b30de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13205) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13213)
* d6167ff53f72e8b630dd794c9d1568026e58fba1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13308)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] Could have duplicates if HoodieRealtimePath has log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1322112229
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ccbbfbae60ff01e765d89add71ba7382e623c5b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
codope merged PR #7264:
URL: https://github.com/apache/hudi/pull/7264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on a diff in pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
boneanxs commented on code in PR #7264:
URL: https://github.com/apache/hudi/pull/7264#discussion_r1034337202
##########
hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieMergeOnReadTableInputFormat.java:
##########
@@ -65,4 +66,16 @@ void pathNotSplitableForBootstrapScenario() throws IOException {
rtPath.setPathWithBootstrapFileStatus(path);
assertFalse(new HoodieMergeOnReadTableInputFormat().isSplitable(fs, rtPath), "Path for bootstrap should not be splitable.");
}
+
+ @Test
+ void pathNotSplitableIfContainsDeltaFiles() throws IOException {
+ URI basePath = Files.createTempFile(tempDir, "target", ".parquet").toUri();
+ HoodieRealtimePath rtPath = new HoodieRealtimePath(new Path("foo"), "bar", basePath.toString(), Collections.emptyList(), "000", false, Option.empty());
+ assertTrue(new HoodieMergeOnReadTableInputFormat().isSplitable(fs, rtPath));
+
+ URI logPath = Files.createTempFile(tempDir, ".test", ".log.4_1-149-180").toUri();
+ HoodieLogFile logFile = new HoodieLogFile(fs.getFileStatus(new Path(logPath)));
+ rtPath = new HoodieRealtimePath(new Path("foo"), "bar", basePath.toString(), Collections.singletonList(logFile), "000", false, Option.empty());
+ assertFalse(new HoodieMergeOnReadTableInputFormat().isSplitable(fs, rtPath), "Path for bootstrap should not be splitable.");
+ }
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1323646088
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177",
"triggerID" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b07f271c16acd022d7853df55bb0a838b8b30de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1323427163
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ccbbfbae60ff01e765d89add71ba7382e623c5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154)
* 1b07f271c16acd022d7853df55bb0a838b8b30de UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] Could have duplicates if HoodieRealtimePath has log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1322445535
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ccbbfbae60ff01e765d89add71ba7382e623c5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1325073746
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177",
"triggerID" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13205",
"triggerID" : "1324450660",
"triggerType" : "MANUAL"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13213",
"triggerID" : "1324651032",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1b07f271c16acd022d7853df55bb0a838b8b30de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13205) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13213)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
boneanxs commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1324651032
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1330143854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177",
"triggerID" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13205",
"triggerID" : "1324450660",
"triggerType" : "MANUAL"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13213",
"triggerID" : "1324651032",
"triggerType" : "MANUAL"
}, {
"hash" : "d6167ff53f72e8b630dd794c9d1568026e58fba1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d6167ff53f72e8b630dd794c9d1568026e58fba1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b07f271c16acd022d7853df55bb0a838b8b30de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13205) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13213)
* d6167ff53f72e8b630dd794c9d1568026e58fba1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1330502039
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177",
"triggerID" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13205",
"triggerID" : "1324450660",
"triggerType" : "MANUAL"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13213",
"triggerID" : "1324651032",
"triggerType" : "MANUAL"
}, {
"hash" : "d6167ff53f72e8b630dd794c9d1568026e58fba1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13308",
"triggerID" : "d6167ff53f72e8b630dd794c9d1568026e58fba1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6167ff53f72e8b630dd794c9d1568026e58fba1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13308)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7264: [HUDI-5253] Could have duplicates if HoodieRealtimePath has log files
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7264:
URL: https://github.com/apache/hudi/pull/7264#discussion_r1028932233
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieRealtimePath.java:
##########
@@ -89,7 +89,7 @@ public boolean getBelongsToIncrementalQuery() {
}
public boolean isSplitable() {
- return !toString().contains(".log") && !includeBootstrapFilePath();
+ return !toString().contains(".log") && !deltaLogFiles.isEmpty() && !includeBootstrapFilePath();
}
Review Comment:
empty or not empty ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1324472586
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13154",
"triggerID" : "7ccbbfbae60ff01e765d89add71ba7382e623c5b",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177",
"triggerID" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"triggerType" : "PUSH"
}, {
"hash" : "1b07f271c16acd022d7853df55bb0a838b8b30de",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13205",
"triggerID" : "1324450660",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1b07f271c16acd022d7853df55bb0a838b8b30de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13177) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13205)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7264:
URL: https://github.com/apache/hudi/pull/7264#discussion_r1034244414
##########
hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieMergeOnReadTableInputFormat.java:
##########
@@ -65,4 +66,16 @@ void pathNotSplitableForBootstrapScenario() throws IOException {
rtPath.setPathWithBootstrapFileStatus(path);
assertFalse(new HoodieMergeOnReadTableInputFormat().isSplitable(fs, rtPath), "Path for bootstrap should not be splitable.");
}
+
+ @Test
+ void pathNotSplitableIfContainsDeltaFiles() throws IOException {
+ URI basePath = Files.createTempFile(tempDir, "target", ".parquet").toUri();
+ HoodieRealtimePath rtPath = new HoodieRealtimePath(new Path("foo"), "bar", basePath.toString(), Collections.emptyList(), "000", false, Option.empty());
+ assertTrue(new HoodieMergeOnReadTableInputFormat().isSplitable(fs, rtPath));
+
+ URI logPath = Files.createTempFile(tempDir, ".test", ".log.4_1-149-180").toUri();
+ HoodieLogFile logFile = new HoodieLogFile(fs.getFileStatus(new Path(logPath)));
+ rtPath = new HoodieRealtimePath(new Path("foo"), "bar", basePath.toString(), Collections.singletonList(logFile), "000", false, Option.empty());
+ assertFalse(new HoodieMergeOnReadTableInputFormat().isSplitable(fs, rtPath), "Path for bootstrap should not be splitable.");
+ }
Review Comment:
> Path for bootstrap should not be splitable
Is the error message right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7264:
URL: https://github.com/apache/hudi/pull/7264#discussion_r1034244582
##########
hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieMergeOnReadTableInputFormat.java:
##########
@@ -65,4 +66,16 @@ void pathNotSplitableForBootstrapScenario() throws IOException {
rtPath.setPathWithBootstrapFileStatus(path);
assertFalse(new HoodieMergeOnReadTableInputFormat().isSplitable(fs, rtPath), "Path for bootstrap should not be splitable.");
}
+
+ @Test
+ void pathNotSplitableIfContainsDeltaFiles() throws IOException {
+ URI basePath = Files.createTempFile(tempDir, "target", ".parquet").toUri();
+ HoodieRealtimePath rtPath = new HoodieRealtimePath(new Path("foo"), "bar", basePath.toString(), Collections.emptyList(), "000", false, Option.empty());
+ assertTrue(new HoodieMergeOnReadTableInputFormat().isSplitable(fs, rtPath));
Review Comment:
Supplement the error message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on pull request #7264: [HUDI-5253] HoodieMergeOnReadTableInputFormat could have duplicate records issue if it contains delta files while still splittable
Posted by GitBox <gi...@apache.org>.
boneanxs commented on PR #7264:
URL: https://github.com/apache/hudi/pull/7264#issuecomment-1324450660
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org