You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/13 10:22:09 UTC
[GitHub] [hudi] Aiden-Dong opened a new pull request, #6664: [HUDI-4308] READ_OPTIMIZED 读取模式在压缩时会暂时丢失数据
Aiden-Dong opened a new pull request, #6664:
URL: https://github.com/apache/hudi/pull/6664
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
**Risk level: none | low | medium | high**
_Choose one. If medium or high, explain what verification was done to mitigate the risks._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
yihua commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261818074
@Aiden-Dong I've updated the PR. You can verify if this solves the problem on your side as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261832232
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
}, {
"hash" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338",
"triggerID" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860",
"triggerID" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11880",
"triggerID" : "1261822206",
"triggerType" : "MANUAL"
}, {
"hash" : "0e1fd58e12e85ebe25548d6637304565b79a2c60",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11882",
"triggerID" : "0e1fd58e12e85ebe25548d6637304565b79a2c60",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7e761ab1c54b0932bdd9e53f5f64d86f15088491 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11880)
* 0e1fd58e12e85ebe25548d6637304565b79a2c60 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11882)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261974532
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
}, {
"hash" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338",
"triggerID" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860",
"triggerID" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11880",
"triggerID" : "1261822206",
"triggerType" : "MANUAL"
}, {
"hash" : "0e1fd58e12e85ebe25548d6637304565b79a2c60",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11882",
"triggerID" : "0e1fd58e12e85ebe25548d6637304565b79a2c60",
"triggerType" : "PUSH"
}, {
"hash" : "0e1fd58e12e85ebe25548d6637304565b79a2c60",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11888",
"triggerID" : "1261938186",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 7e761ab1c54b0932bdd9e53f5f64d86f15088491 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11880)
* 0e1fd58e12e85ebe25548d6637304565b79a2c60 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11882) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11888)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261503351
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
}, {
"hash" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338",
"triggerID" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860",
"triggerID" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 01ee0a18ee96df248c3cde71efb39a1a08a33518 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338)
* 7e761ab1c54b0932bdd9e53f5f64d86f15088491 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1246207480
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
}, {
"hash" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338",
"triggerID" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 01ee0a18ee96df248c3cde71efb39a1a08a33518 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6664:
URL: https://github.com/apache/hudi/pull/6664#discussion_r983118499
##########
hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java:
##########
@@ -291,31 +292,23 @@ private void doRefresh() {
validate(activeTimeline, queryInstant);
- if (tableType.equals(HoodieTableType.MERGE_ON_READ) && queryType.equals(HoodieTableQueryType.SNAPSHOT)) {
- cachedAllInputFileSlices = partitionFiles.keySet().stream()
- .collect(Collectors.toMap(
- Function.identity(),
- partitionPath ->
- queryInstant.map(instant ->
- fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
+ // NOTE: For MOR table, when the compaction is inflight, we need to not only fetch the
+ // latest slices, but also include the base and log files of the second-last version of
+ // the file slice in the same file group as the latest file slice that is under compaction.
+ // This logic is realized by `AbstractTableFileSystemView::getLatestMergedFileSlicesBeforeOrOn`
+ // API. Note that for COW table, the merging logic of two slices does not happen as there
+ // is no compaction, thus there is no performance impact.
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+ fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
Review Comment:
can you add docs around how this single api would suffice for all 3 callers (COW, MOR Ro, MOR RT).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6664:
URL: https://github.com/apache/hudi/pull/6664#discussion_r983020758
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala:
##########
@@ -1128,12 +1128,14 @@ class TestHoodieSparkSqlWriter {
new JavaSparkContext(sc), "", table_path, hoodieFooTableName,
mapAsJavaMap(compactionOperation)).asInstanceOf[SparkRDDWriteClient[HoodieRecordPayload[Nothing]]]
- client.scheduleCompaction(org.apache.hudi.common.util.Option.empty())
- client.close()
+ val compactionInstant = client.scheduleCompaction(org.apache.hudi.common.util.Option.empty()).get()
+ client.compact(compactionInstant)
Review Comment:
@Aiden-Dong Thanks for raising this. Actually, `client.compact()` executes the compaction by writing the new base files without committing to the timeline of the table, i.e., the compaction instant is still inflight in the timeline. So this is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261548112
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
}, {
"hash" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338",
"triggerID" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860",
"triggerID" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7e761ab1c54b0932bdd9e53f5f64d86f15088491 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Aiden-Dong commented on a diff in pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
Aiden-Dong commented on code in PR #6664:
URL: https://github.com/apache/hudi/pull/6664#discussion_r982989501
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala:
##########
@@ -1128,12 +1128,14 @@ class TestHoodieSparkSqlWriter {
new JavaSparkContext(sc), "", table_path, hoodieFooTableName,
mapAsJavaMap(compactionOperation)).asInstanceOf[SparkRDDWriteClient[HoodieRecordPayload[Nothing]]]
- client.scheduleCompaction(org.apache.hudi.common.util.Option.empty())
- client.close()
+ val compactionInstant = client.scheduleCompaction(org.apache.hudi.common.util.Option.empty()).get()
+ client.compact(compactionInstant)
Review Comment:
client.compact(compactionInstant) cannot be executed here, or the compression will be completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Aiden-Dong commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
Aiden-Dong commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261938186
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1245249473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11700a2f56248c9db681190f89c259546242edf1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261982163
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
}, {
"hash" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338",
"triggerID" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860",
"triggerID" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11880",
"triggerID" : "1261822206",
"triggerType" : "MANUAL"
}, {
"hash" : "0e1fd58e12e85ebe25548d6637304565b79a2c60",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11882",
"triggerID" : "0e1fd58e12e85ebe25548d6637304565b79a2c60",
"triggerType" : "PUSH"
}, {
"hash" : "0e1fd58e12e85ebe25548d6637304565b79a2c60",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11888",
"triggerID" : "1261938186",
"triggerType" : "MANUAL"
}, {
"hash" : "0e1fd58e12e85ebe25548d6637304565b79a2c60",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11882",
"triggerID" : "1261938186",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0e1fd58e12e85ebe25548d6637304565b79a2c60 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11888) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11882)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1246126933
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
}, {
"hash" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11700a2f56248c9db681190f89c259546242edf1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324)
* 01ee0a18ee96df248c3cde71efb39a1a08a33518 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Aiden-Dong commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
Aiden-Dong commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1246122270
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Aiden-Dong commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
Aiden-Dong commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1246383184
@yihua can you help for review this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261823604
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
}, {
"hash" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338",
"triggerID" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860",
"triggerID" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11880",
"triggerID" : "1261822206",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 7e761ab1c54b0932bdd9e53f5f64d86f15088491 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11880)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6664:
URL: https://github.com/apache/hudi/pull/6664#discussion_r982746433
##########
hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java:
##########
@@ -304,6 +304,18 @@ private void doRefresh() {
.orElse(Collections.emptyList())
)
);
+ } else if (tableType.equals(HoodieTableType.MERGE_ON_READ) && queryType.equals(HoodieTableQueryType.READ_OPTIMIZED)) {
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+ fileSystemView.getLatestMergedFileSlicesWithOnlyBaseFileBeforeOrOn(partitionPath.path, queryInstant.get())
Review Comment:
Basically, you're fixing the problem where the compaction is in progress and the file slice falling into the compaction under construction should not be included until it is committed. Shall we fix the `getLatestFileSlicesBeforeOrOn` instead of adding a new API here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
yihua commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261688993
Hi @Aiden-Dong, thanks for raising the issue and the potential fix! Inspired by that, I identified the root cause. I'm going to update the PR soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
yihua merged PR #6664:
URL: https://github.com/apache/hudi/pull/6664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1246129576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
}, {
"hash" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338",
"triggerID" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11700a2f56248c9db681190f89c259546242edf1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324)
* 01ee0a18ee96df248c3cde71efb39a1a08a33518 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1246124276
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 11700a2f56248c9db681190f89c259546242edf1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1245325224
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11700a2f56248c9db681190f89c259546242edf1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6664:
URL: https://github.com/apache/hudi/pull/6664#discussion_r983713186
##########
hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java:
##########
@@ -291,31 +292,23 @@ private void doRefresh() {
validate(activeTimeline, queryInstant);
- if (tableType.equals(HoodieTableType.MERGE_ON_READ) && queryType.equals(HoodieTableQueryType.SNAPSHOT)) {
- cachedAllInputFileSlices = partitionFiles.keySet().stream()
- .collect(Collectors.toMap(
- Function.identity(),
- partitionPath ->
- queryInstant.map(instant ->
- fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
+ // NOTE: For MOR table, when the compaction is inflight, we need to not only fetch the
+ // latest slices, but also include the base and log files of the second-last version of
+ // the file slice in the same file group as the latest file slice that is under compaction.
+ // This logic is realized by `AbstractTableFileSystemView::getLatestMergedFileSlicesBeforeOrOn`
+ // API. Note that for COW table, the merging logic of two slices does not happen as there
+ // is no compaction, thus there is no performance impact.
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+ fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
Review Comment:
I added the docs above. Is that sufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
yihua commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1262492512
CI passes.
<img width="1608" alt="Screen Shot 2022-09-29 at 09 03 42" src="https://user-images.githubusercontent.com/2497195/193081804-9c395e75-ea55-4a23-9897-517944e896f7.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
yihua commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261822206
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1245255216
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11700a2f56248c9db681190f89c259546242edf1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261827848
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
}, {
"hash" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338",
"triggerID" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860",
"triggerID" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11880",
"triggerID" : "1261822206",
"triggerType" : "MANUAL"
}, {
"hash" : "0e1fd58e12e85ebe25548d6637304565b79a2c60",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0e1fd58e12e85ebe25548d6637304565b79a2c60",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7e761ab1c54b0932bdd9e53f5f64d86f15088491 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11860) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11880)
* 0e1fd58e12e85ebe25548d6637304565b79a2c60 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Aiden-Dong commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
Aiden-Dong commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261692225
> Hi @Aiden-Dong, thanks for raising the issue and the potential fix! Inspired by that, I identified the root cause. I'm going to update the PR soon.
thanks for your answer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Aiden-Dong commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
Aiden-Dong commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261852143
>
tks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6664: [HUDI-4308] READ_OPTIMIZED read mode will temporary loss of data when compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6664:
URL: https://github.com/apache/hudi/pull/6664#issuecomment-1261499339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "11700a2f56248c9db681190f89c259546242edf1",
"triggerType" : "PUSH"
}, {
"hash" : "11700a2f56248c9db681190f89c259546242edf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11324",
"triggerID" : "1246122270",
"triggerType" : "MANUAL"
}, {
"hash" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338",
"triggerID" : "01ee0a18ee96df248c3cde71efb39a1a08a33518",
"triggerType" : "PUSH"
}, {
"hash" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7e761ab1c54b0932bdd9e53f5f64d86f15088491",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 01ee0a18ee96df248c3cde71efb39a1a08a33518 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11338)
* 7e761ab1c54b0932bdd9e53f5f64d86f15088491 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org