You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/01/08 13:20:17 UTC
[GitHub] [hudi] YuweiXiao opened a new pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
YuweiXiao opened a new pull request #4540:
URL: https://github.com/apache/hudi/pull/4540
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Fix MOR snapshot query path during compaction for HIVE read.
In current implementation, if a write comes in and complete during compaction, it will not be visible to snapshot query until the compaction completes. This is caused by filter logic of getting file group's log files.
## Brief change log
- Modify the file group retrieval logic in MOR snapshot query path, i.e., `HoodieInputFormatUtils::filterFileStatusForSnapshotMode`
## Verify this pull request
This change added tests and can be verified as follows:
- Add functional test about snapshot query during compaction.
## Committer checklist
- [x] Has a corresponding JIRA in PR title & commit
- [x] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008636446
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
}, {
"hash" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048",
"triggerID" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 52cad3508ddf12c73f1c5c60180fe1137232192d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008501296
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc6e817b518774152944d658e4c239cfcce30c9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016)
* c3295aa79ecd15281ffc573c86e73a2637f3533f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008006702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc6e817b518774152944d658e4c239cfcce30c9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008604559
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
}, {
"hash" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048",
"triggerID" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3295aa79ecd15281ffc573c86e73a2637f3533f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041)
* 52cad3508ddf12c73f1c5c60180fe1137232192d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008343888
@YuweiXiao : I see "WIP" in title. Is the patch good to review or is it still being worked upon ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008006702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc6e817b518774152944d658e4c239cfcce30c9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008698046
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
}, {
"hash" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048",
"triggerID" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"triggerType" : "PUSH"
}, {
"hash" : "49b07964078f73338ab7c9bd7c1bcc67211fecdd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5051",
"triggerID" : "49b07964078f73338ab7c9bd7c1bcc67211fecdd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49b07964078f73338ab7c9bd7c1bcc67211fecdd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5051)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008604559
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
}, {
"hash" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048",
"triggerID" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3295aa79ecd15281ffc573c86e73a2637f3533f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041)
* 52cad3508ddf12c73f1c5c60180fe1137232192d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008653366
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
}, {
"hash" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048",
"triggerID" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"triggerType" : "PUSH"
}, {
"hash" : "49b07964078f73338ab7c9bd7c1bcc67211fecdd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5051",
"triggerID" : "49b07964078f73338ab7c9bd7c1bcc67211fecdd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 52cad3508ddf12c73f1c5c60180fe1137232192d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048)
* 49b07964078f73338ab7c9bd7c1bcc67211fecdd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5051)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008501965
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc6e817b518774152944d658e4c239cfcce30c9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016)
* c3295aa79ecd15281ffc573c86e73a2637f3533f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008532182
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3295aa79ecd15281ffc573c86e73a2637f3533f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008603238
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
}, {
"hash" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3295aa79ecd15281ffc573c86e73a2637f3533f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041)
* 52cad3508ddf12c73f1c5c60180fe1137232192d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao edited a comment on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
YuweiXiao edited a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1009560576
> @YuweiXiao only a small question: Does clustering also have this problem？
I guess not. Currently, clustering doesn't support concurrent updates. So there won't be new log files when doing the clustering.
But you remind me it may be a problem in the future. I am working on consistent hashing index recently, which could enhance clustering to support concurrent update. I will keep an eye on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1009530773
> @YuweiXiao Why not use `BaseFileWithLogsSplit` which maps matching log files to base files?
Actually, I tried with `BaseFileWithLogsSplit`. IIUC, we need to use `BaseFileWithLogsSplit` at the place where we generate the split, i.e., `HoodieInputFormatUtils::filterFileStatusForSnapshotMode`. By looking at its implementation, I found its semantic is to generate baseFile split and `log file only` file slice. And callers of this function rely on this semantic, such as bootstrap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on a change in pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on a change in pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#discussion_r781702301
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -94,16 +94,16 @@
HoodieTableMetaClient metaClient = partitionsToMetaClient.get(partitionPath);
if (!fsCache.containsKey(metaClient)) {
HoodieLocalEngineContext engineContext = new HoodieLocalEngineContext(conf);
- HoodieTableFileSystemView fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
- metaClient, HoodieInputFormatUtils.buildMetadataConfig(conf));
+ HoodieTableFileSystemView fsView = FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext,
+ metaClient, HoodieInputFormatUtils.buildMetadataConfig(conf), metaClient.getActiveTimeline());
fsCache.put(metaClient, fsView);
}
HoodieTableFileSystemView fsView = fsCache.get(metaClient);
String relPartitionPath = FSUtils.getRelativePartitionPath(new Path(metaClient.getBasePath()), partitionPath);
// Both commit and delta-commits are included - pick the latest completed one
Option<HoodieInstant> latestCompletedInstant =
- metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ metaClient.getActiveTimeline().getWriteTimeline().filterCompletedInstants().lastInstant();
Review comment:
It won't affect the correctness. The `latestCompletedInstant` is used to filter file slice. Considering a compaction only case, without including the completed compaction instant, we will end up reading 'old version' file slice (i.e., base file + log) rather than the compacted one (i.e., only base file, which has better performance).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1009560576
> @YuweiXiao only a small question: Does clustering also have this problem？
I guess not. Currently, clustering doesn't support concurrent updates. So there won't be new log files when doing the clustering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1009559883
@YuweiXiao only a small question: Does clustering also have this problem？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008636446
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
}, {
"hash" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048",
"triggerID" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 52cad3508ddf12c73f1c5c60180fe1137232192d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008479468
> @YuweiXiao : I see "WIP" in title. Is the patch good to review or is it still being worked upon ?
Yes, it is still in process, as there are failed UT need to be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008653366
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
}, {
"hash" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048",
"triggerID" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"triggerType" : "PUSH"
}, {
"hash" : "49b07964078f73338ab7c9bd7c1bcc67211fecdd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5051",
"triggerID" : "49b07964078f73338ab7c9bd7c1bcc67211fecdd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 52cad3508ddf12c73f1c5c60180fe1137232192d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048)
* 49b07964078f73338ab7c9bd7c1bcc67211fecdd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5051)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008501296
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc6e817b518774152944d658e4c239cfcce30c9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016)
* c3295aa79ecd15281ffc573c86e73a2637f3533f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008603238
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
}, {
"hash" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3295aa79ecd15281ffc573c86e73a2637f3533f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041)
* 52cad3508ddf12c73f1c5c60180fe1137232192d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#discussion_r781314302
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -94,16 +94,16 @@
HoodieTableMetaClient metaClient = partitionsToMetaClient.get(partitionPath);
if (!fsCache.containsKey(metaClient)) {
HoodieLocalEngineContext engineContext = new HoodieLocalEngineContext(conf);
- HoodieTableFileSystemView fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
- metaClient, HoodieInputFormatUtils.buildMetadataConfig(conf));
+ HoodieTableFileSystemView fsView = FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext,
+ metaClient, HoodieInputFormatUtils.buildMetadataConfig(conf), metaClient.getActiveTimeline());
fsCache.put(metaClient, fsView);
}
HoodieTableFileSystemView fsView = fsCache.get(metaClient);
String relPartitionPath = FSUtils.getRelativePartitionPath(new Path(metaClient.getBasePath()), partitionPath);
// Both commit and delta-commits are included - pick the latest completed one
Option<HoodieInstant> latestCompletedInstant =
- metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ metaClient.getActiveTimeline().getWriteTimeline().filterCompletedInstants().lastInstant();
Review comment:
The writeTimeline will also contain the compaction instant compared to commitsTimeline, but how does that matter for this scenario? Since latest active timeline is already being passed to `createInMemoryFileSystemViewWithTimeline` then latest file slice would contain the file group due to commit during ongoing compaction right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1009573278
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#discussion_r784430762
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -94,16 +94,16 @@
HoodieTableMetaClient metaClient = partitionsToMetaClient.get(partitionPath);
if (!fsCache.containsKey(metaClient)) {
HoodieLocalEngineContext engineContext = new HoodieLocalEngineContext(conf);
- HoodieTableFileSystemView fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
- metaClient, HoodieInputFormatUtils.buildMetadataConfig(conf));
+ HoodieTableFileSystemView fsView = FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext,
+ metaClient, HoodieInputFormatUtils.buildMetadataConfig(conf), metaClient.getActiveTimeline());
fsCache.put(metaClient, fsView);
}
HoodieTableFileSystemView fsView = fsCache.get(metaClient);
String relPartitionPath = FSUtils.getRelativePartitionPath(new Path(metaClient.getBasePath()), partitionPath);
// Both commit and delta-commits are included - pick the latest completed one
Option<HoodieInstant> latestCompletedInstant =
- metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ metaClient.getActiveTimeline().getWriteTimeline().filterCompletedInstants().lastInstant();
Review comment:
got it. thanks for explaining in detail. Fix makes sense then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#discussion_r783918083
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -94,16 +94,16 @@
HoodieTableMetaClient metaClient = partitionsToMetaClient.get(partitionPath);
if (!fsCache.containsKey(metaClient)) {
HoodieLocalEngineContext engineContext = new HoodieLocalEngineContext(conf);
- HoodieTableFileSystemView fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
- metaClient, HoodieInputFormatUtils.buildMetadataConfig(conf));
+ HoodieTableFileSystemView fsView = FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext,
+ metaClient, HoodieInputFormatUtils.buildMetadataConfig(conf), metaClient.getActiveTimeline());
fsCache.put(metaClient, fsView);
}
HoodieTableFileSystemView fsView = fsCache.get(metaClient);
String relPartitionPath = FSUtils.getRelativePartitionPath(new Path(metaClient.getBasePath()), partitionPath);
// Both commit and delta-commits are included - pick the latest completed one
Option<HoodieInstant> latestCompletedInstant =
- metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ metaClient.getActiveTimeline().getWriteTimeline().filterCompletedInstants().lastInstant();
Review comment:
I also don't understand the fix. can you help throw some light. From the description in this patch, the gap is, when compaction is on-going and a new write comes in and completes, it may not be visible to queries.
But the fix here, just includes compaction instants to the list of instants to process. Not sure if the description matches the fix.
or am I missing anything here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1007988388
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc6e817b518774152944d658e4c239cfcce30c9f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1007988827
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc6e817b518774152944d658e4c239cfcce30c9f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1007988827
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc6e817b518774152944d658e4c239cfcce30c9f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008535743
@xiarixiaoyao : hey, can you review this patch please. Touches part of the code authored by you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008650816
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
}, {
"hash" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048",
"triggerID" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"triggerType" : "PUSH"
}, {
"hash" : "49b07964078f73338ab7c9bd7c1bcc67211fecdd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49b07964078f73338ab7c9bd7c1bcc67211fecdd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 52cad3508ddf12c73f1c5c60180fe1137232192d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048)
* 49b07964078f73338ab7c9bd7c1bcc67211fecdd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008650816
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
}, {
"hash" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048",
"triggerID" : "52cad3508ddf12c73f1c5c60180fe1137232192d",
"triggerType" : "PUSH"
}, {
"hash" : "49b07964078f73338ab7c9bd7c1bcc67211fecdd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49b07964078f73338ab7c9bd7c1bcc67211fecdd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 52cad3508ddf12c73f1c5c60180fe1137232192d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5048)
* 49b07964078f73338ab7c9bd7c1bcc67211fecdd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1011081051
@codope could you pls review this pr again, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1007988388
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc6e817b518774152944d658e4c239cfcce30c9f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008532182
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3295aa79ecd15281ffc573c86e73a2637f3533f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4540: [HUDI-3194][WIP] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#issuecomment-1008501965
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016",
"triggerID" : "dc6e817b518774152944d658e4c239cfcce30c9f",
"triggerType" : "PUSH"
}, {
"hash" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041",
"triggerID" : "c3295aa79ecd15281ffc573c86e73a2637f3533f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc6e817b518774152944d658e4c239cfcce30c9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5016)
* c3295aa79ecd15281ffc573c86e73a2637f3533f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5041)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on a change in pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on a change in pull request #4540:
URL: https://github.com/apache/hudi/pull/4540#discussion_r783929268
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -94,16 +94,16 @@
HoodieTableMetaClient metaClient = partitionsToMetaClient.get(partitionPath);
if (!fsCache.containsKey(metaClient)) {
HoodieLocalEngineContext engineContext = new HoodieLocalEngineContext(conf);
- HoodieTableFileSystemView fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
- metaClient, HoodieInputFormatUtils.buildMetadataConfig(conf));
+ HoodieTableFileSystemView fsView = FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext,
+ metaClient, HoodieInputFormatUtils.buildMetadataConfig(conf), metaClient.getActiveTimeline());
fsCache.put(metaClient, fsView);
}
HoodieTableFileSystemView fsView = fsCache.get(metaClient);
String relPartitionPath = FSUtils.getRelativePartitionPath(new Path(metaClient.getBasePath()), partitionPath);
// Both commit and delta-commits are included - pick the latest completed one
Option<HoodieInstant> latestCompletedInstant =
- metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ metaClient.getActiveTimeline().getWriteTimeline().filterCompletedInstants().lastInstant();
Review comment:
> I also don't understand the fix. can you help throw some light. From the description in this patch, the gap is, when compaction is on-going and a new write comes in and completes, it may not be visible to queries. But the fix here, just includes compaction instants to the list of instants to process. Not sure if the description matches the fix. or am I missing anything here.
Hey! In `fsView::getLatestMergedFileSlicesBeforeOrOn`, there is a logic where we check if a file group is under compaction (under construction), so that we could add logs files generated by concurrent writers. And only passing a timeline including compactions, this logic could work (`fsView::fetchMergedFileSlice`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #4540: [HUDI-3194] fix MOR snapshot query (HIVE) during compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #4540:
URL: https://github.com/apache/hudi/pull/4540
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org