You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/01/22 03:25:28 UTC
[GitHub] [hudi] alexeykudinkin opened a new pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
alexeykudinkin opened a new pull request #4669:
URL: https://github.com/apache/hudi/pull/4669
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Converting `BaseHoodieTableFileIndex` to Java, removing Scala as a dependency from "hudi-common"
## Brief change log
See above
## Verify this pull request
This pull request is already covered by existing tests, such as *(please describe tests)*.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020641005
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89266f9e490e4129431d15ac4c7103eb799540e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429)
* f01e128350dd5910ddc51865303df29170317ac0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020696601
@alexeykudinkin Thanks for your contribution! Can you figure out why CI failed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1033265778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814",
"triggerID" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c7024e7f5e09b48b4da580a4fed0c458a566d99 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777)
* dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r799903374
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
This has been introduced in one of the PRs in this stack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #4669:
URL: https://github.com/apache/hudi/pull/4669
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r803272577
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
@nsivabalan looks like we are good here. These are totally new enums, not there in 0.10.0 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1033228002
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c7024e7f5e09b48b4da580a4fed0c458a566d99 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777)
* dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1031884934
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d12ce874c3bfb9861b00449b90ebde0125388b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715)
* 9c7024e7f5e09b48b4da580a4fed0c458a566d99 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1029465605
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54e68ec73aa7292556d5f00c90859a776acc109f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687)
* 7d12ce874c3bfb9861b00449b90ebde0125388b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1029517702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d12ce874c3bfb9861b00449b90ebde0125388b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020828978
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 851e61cdb31748501d28d638bc192ccc4955d665 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020643261
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89266f9e490e4129431d15ac4c7103eb799540e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429)
* f01e128350dd5910ddc51865303df29170317ac0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020687055
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f01e128350dd5910ddc51865303df29170317ac0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802123222
##########
File path: hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class BaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCache fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCache fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ doRefresh();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> getLatestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ doRefresh();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
Review comment:
in master I see here we are using fullPartitionPathsMapToFetch.values.toArray() ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class BaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCache fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCache fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ doRefresh();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> getLatestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ doRefresh();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void doRefresh() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.or(() -> latestInstant.map(HoodieInstant::getTimestamp));
Review comment:
what happens if latestInstant is empty ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class BaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCache fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCache fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ doRefresh();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> getLatestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ doRefresh();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
Review comment:
we can instantiate to empty map here only. and later follow it as
```
if (!pathToFetch.isEmpty()) {) {
// update
}
```
or was it intentionally written this way ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
I see deduceQueryType in SparkHoodieTableFileIndex. so if we are converting the old vars to new ones, we should be good. Can you do some sanity testing w/ quick start guide that old variables work w/ this patch too.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
I am not sure if we can do this. Users might have to fix all of their existing pipelines w/ these new variables. atleast for read optimized query and incremental query pipelines.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
and does the variable conversion happen in all code paths. for eg, there is a config where in you can disable fileIndex altogether. So, presto, trino, hive connector, beeline, spark datasource from all possible paths, ensure old to new is taken care of.
##########
File path: hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class BaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCache fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCache fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ doRefresh();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> getLatestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ doRefresh();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
Review comment:
oh, we are reverting the key and value in fullPartitionPathsMapToFetch. got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r801033300
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
Review comment:
Totally
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
Review comment:
Could be. Why you think it's better to be List?
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
Review comment:
Not sure i follow your suggestion here. Can you elaborate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802086671
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ refresh0();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void refresh0() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.orElseOption(latestInstant.map(HoodieInstant::getTimestamp));
+
+ if (tableType.equals(HoodieTableType.MERGE_ON_READ) && queryType.equals(HoodieTableQueryType.SNAPSHOT)) {
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+ fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
+ .collect(Collectors.toList())
+ )
+ .orElse(Collections.emptyList())
+ )
+ );
+ } else {
+ // TODO re-align with the branch (MOR, snapshot) branch
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ specifiedQueryInstant.map(instant ->
+ fileSystemView.getLatestFileSlicesBeforeOrOn(partitionPath.path, instant, true))
+ .orElse(fileSystemView.getLatestFileSlices(partitionPath.path))
+ .collect(Collectors.toList())
+ )
+ );
+ }
+
+ cachedFileSize = cachedAllInputFileSlices.values().stream()
+ .flatMap(Collection::stream)
+ .mapToLong(JavaBaseHoodieTableFileIndex::fileSliceSize)
+ .sum();
+
+ // If the partition value contains InternalRow.empty, we query it as a non-partitioned table.
+ queryAsNonePartitionedTable = partitionFiles.keySet().stream().anyMatch(p -> p.values.length == 0);
+
+ long duration = System.currentTimeMillis() - startTime;
+
+ LOG.info(String.format("Refresh table %s, spent: %d ms", metaClient.getTableConfig().getTableName(), duration));
+ }
+
+ private static long fileSliceSize(FileSlice fileSlice) {
+ long logFileSize = fileSlice.getLogFiles().map(HoodieLogFile::getFileSize)
+ .filter(s -> s > 0)
+ .reduce(0L, Long::sum);
+
+ return fileSlice.getBaseFile().map(BaseFile::getFileLen).orElse(0L) + logFileSize;
+ }
+
+ protected static final class PartitionPath {
+ final String path;
+ final Object[] values;
+
+ public PartitionPath(String path, Object[] values) {
+ this.path = path;
+ this.values = values;
+ }
+
+ Path fullPartitionPath(String basePath) {
+ if (!path.isEmpty()) {
+ return new Path(basePath, path);
+ }
+
+ return new Path(basePath);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof PartitionPath
+ && Objects.equals(path, ((PartitionPath) other).path)
+ && Arrays.equals(values, ((PartitionPath) other).values);
Review comment:
Sg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r800923920
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ refresh0();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void refresh0() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.orElseOption(latestInstant.map(HoodieInstant::getTimestamp));
+
+ if (tableType.equals(HoodieTableType.MERGE_ON_READ) && queryType.equals(HoodieTableQueryType.SNAPSHOT)) {
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+ fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
+ .collect(Collectors.toList())
+ )
+ .orElse(Collections.emptyList())
+ )
+ );
+ } else {
+ // TODO re-align with the branch (MOR, snapshot) branch
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ specifiedQueryInstant.map(instant ->
+ fileSystemView.getLatestFileSlicesBeforeOrOn(partitionPath.path, instant, true))
+ .orElse(fileSystemView.getLatestFileSlices(partitionPath.path))
+ .collect(Collectors.toList())
+ )
+ );
+ }
+
+ cachedFileSize = cachedAllInputFileSlices.values().stream()
+ .flatMap(Collection::stream)
+ .mapToLong(JavaBaseHoodieTableFileIndex::fileSliceSize)
+ .sum();
+
+ // If the partition value contains InternalRow.empty, we query it as a non-partitioned table.
+ queryAsNonePartitionedTable = partitionFiles.keySet().stream().anyMatch(p -> p.values.length == 0);
+
+ long duration = System.currentTimeMillis() - startTime;
+
+ LOG.info(String.format("Refresh table %s, spent: %d ms", metaClient.getTableConfig().getTableName(), duration));
+ }
+
+ private static long fileSliceSize(FileSlice fileSlice) {
+ long logFileSize = fileSlice.getLogFiles().map(HoodieLogFile::getFileSize)
+ .filter(s -> s > 0)
+ .reduce(0L, Long::sum);
+
+ return fileSlice.getBaseFile().map(BaseFile::getFileLen).orElse(0L) + logFileSize;
Review comment:
Shortening the expression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r800923268
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
Review comment:
There's no point in lazy-eval since this class is used eagerly in the flow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1026300648
@yanghua there's a long stack of PRs, and there were issues w/ Metadata Table that were failing tests in the PR at the very bottom of it.
These issues have been addressed here in https://github.com/apache/hudi/pull/4716, and as soon as it lands i'll rebase onto it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r799919147
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/Option.java
##########
@@ -123,6 +123,13 @@ public T orElse(T other) {
return val != null ? val : other;
}
+ /**
+ * Returns this {@link Option} if non-empty, and returns the given one otherwise
+ */
+ public Option<T> orElseOption(Option<T> other) {
+ return val != null ? this : other;
+ }
+
Review comment:
Does this provide the same functionality as `public Option<T> or(Supplier<? extends Option<T>> other)` which can be reused?
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
Review comment:
@alexeykudinkin could you remove the `BaseHoodieTableFileIndex.scala` and rename this class to `BaseHoodieTableFileIndex` in this PR? The PR should also include the removal of scala dependency. They belong to the same set of change. It also makes it easier for the community to reason this PR. Other cleanup can stay in the next one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802068021
##########
File path: packaging/hudi-hadoop-mr-bundle/pom.xml
##########
@@ -67,9 +67,6 @@
<include>org.apache.hudi:hudi-common</include>
<include>org.apache.hudi:hudi-hadoop-mr</include>
- <!-- TODO(HUDI-3239) remove this -->
- <include>org.scala-lang:scala-library</include>
-
Review comment:
Should the following dep be removed in this pom as well? Similar for the other bundle poms, `hudi-hadop-mr/pom.xml`,`hudi-common/pom.xml` where scala dep is added in #4531.
```
<!-- Scala -->
<!-- TODO(HUDI-3239) remove this dep -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802083597
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ refresh0();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void refresh0() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.orElseOption(latestInstant.map(HoodieInstant::getTimestamp));
+
+ if (tableType.equals(HoodieTableType.MERGE_ON_READ) && queryType.equals(HoodieTableQueryType.SNAPSHOT)) {
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+ fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
+ .collect(Collectors.toList())
+ )
+ .orElse(Collections.emptyList())
+ )
+ );
+ } else {
+ // TODO re-align with the branch (MOR, snapshot) branch
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ specifiedQueryInstant.map(instant ->
+ fileSystemView.getLatestFileSlicesBeforeOrOn(partitionPath.path, instant, true))
+ .orElse(fileSystemView.getLatestFileSlices(partitionPath.path))
+ .collect(Collectors.toList())
+ )
+ );
+ }
+
+ cachedFileSize = cachedAllInputFileSlices.values().stream()
+ .flatMap(Collection::stream)
+ .mapToLong(JavaBaseHoodieTableFileIndex::fileSliceSize)
+ .sum();
+
+ // If the partition value contains InternalRow.empty, we query it as a non-partitioned table.
+ queryAsNonePartitionedTable = partitionFiles.keySet().stream().anyMatch(p -> p.values.length == 0);
+
+ long duration = System.currentTimeMillis() - startTime;
+
+ LOG.info(String.format("Refresh table %s, spent: %d ms", metaClient.getTableConfig().getTableName(), duration));
+ }
+
+ private static long fileSliceSize(FileSlice fileSlice) {
+ long logFileSize = fileSlice.getLogFiles().map(HoodieLogFile::getFileSize)
+ .filter(s -> s > 0)
+ .reduce(0L, Long::sum);
+
+ return fileSlice.getBaseFile().map(BaseFile::getFileLen).orElse(0L) + logFileSize;
+ }
+
+ protected static final class PartitionPath {
+ final String path;
+ final Object[] values;
+
+ public PartitionPath(String path, Object[] values) {
+ this.path = path;
+ this.values = values;
+ }
+
+ Path fullPartitionPath(String basePath) {
+ if (!path.isEmpty()) {
+ return new Path(basePath, path);
+ }
+
+ return new Path(basePath);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof PartitionPath
+ && Objects.equals(path, ((PartitionPath) other).path)
+ && Arrays.equals(values, ((PartitionPath) other).values);
Review comment:
I was falling back on tests to validate that this doesn't get in the way of any assumption previously made. It's working fine, so i'd rather keep it the way it should be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802102365
##########
File path: hudi-common/pom.xml
##########
@@ -38,18 +38,6 @@
<pluginManagement>
<plugins>
- <plugin>
- <groupId>net.alchim31.maven</groupId>
- <artifactId>scala-maven-plugin</artifactId>
- <version>${scala-maven-plugin.version}</version>
- <configuration>
- <args>
- <arg>-nobootcp</arg>
- <arg>-target:jvm-1.8</arg>
- </args>
- <checkMultipleScalaVersions>false</checkMultipleScalaVersions>
- </configuration>
- </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
Review comment:
Fair enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020641005
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89266f9e490e4129431d15ac4c7103eb799540e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429)
* f01e128350dd5910ddc51865303df29170317ac0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020687055
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f01e128350dd5910ddc51865303df29170317ac0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1019031954
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89266f9e490e4129431d15ac4c7103eb799540e3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1019031954
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89266f9e490e4129431d15ac4c7103eb799540e3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1028370459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 851e61cdb31748501d28d638bc192ccc4955d665 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494)
* 54e68ec73aa7292556d5f00c90859a776acc109f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1026461140
> @yanghua there's a long stack of PRs, and there were issues w/ Metadata Table that were failing tests in the PR at the very bottom of it.
>
> These issues have been addressed here in #4716, and as soon as it lands i'll rebase onto it.
Sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020782050
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f01e128350dd5910ddc51865303df29170317ac0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482)
* 851e61cdb31748501d28d638bc192ccc4955d665 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r799903433
##########
File path: hudi-common/src/main/scala/org/apache/hudi/BaseHoodieTableFileIndex.scala
##########
@@ -51,19 +51,14 @@ import scala.language.implicitConversions
* @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
* @param fileStatusCache transient cache of fetched [[FileStatus]]es
*/
-abstract class HoodieTableFileIndexBase(engineContext: HoodieEngineContext,
+abstract class BaseHoodieTableFileIndex(engineContext: HoodieEngineContext,
Review comment:
Scala one will go away
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1033327239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814",
"triggerID" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802074827
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
Review comment:
I see `queryPaths` is a `List<Path>` so wondering if this can also use `List`. Not an issue though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1033226075
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c7024e7f5e09b48b4da580a4fed0c458a566d99 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802250708
##########
File path: hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class BaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCache fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCache fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ doRefresh();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> getLatestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ doRefresh();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void doRefresh() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.or(() -> latestInstant.map(HoodieInstant::getTimestamp));
Review comment:
here in this expression. don't we need to do OrElse here if incase latestInsant is empty ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1034220069
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020782050
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f01e128350dd5910ddc51865303df29170317ac0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482)
* 851e61cdb31748501d28d638bc192ccc4955d665 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1029467643
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54e68ec73aa7292556d5f00c90859a776acc109f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687)
* 7d12ce874c3bfb9861b00449b90ebde0125388b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020828978
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 851e61cdb31748501d28d638bc192ccc4955d665 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1028372962
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 851e61cdb31748501d28d638bc192ccc4955d665 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494)
* 54e68ec73aa7292556d5f00c90859a776acc109f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r800925073
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ refresh0();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void refresh0() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.orElseOption(latestInstant.map(HoodieInstant::getTimestamp));
+
+ if (tableType.equals(HoodieTableType.MERGE_ON_READ) && queryType.equals(HoodieTableQueryType.SNAPSHOT)) {
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+ fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
+ .collect(Collectors.toList())
+ )
+ .orElse(Collections.emptyList())
+ )
+ );
+ } else {
+ // TODO re-align with the branch (MOR, snapshot) branch
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ specifiedQueryInstant.map(instant ->
+ fileSystemView.getLatestFileSlicesBeforeOrOn(partitionPath.path, instant, true))
+ .orElse(fileSystemView.getLatestFileSlices(partitionPath.path))
+ .collect(Collectors.toList())
+ )
+ );
+ }
+
+ cachedFileSize = cachedAllInputFileSlices.values().stream()
+ .flatMap(Collection::stream)
+ .mapToLong(JavaBaseHoodieTableFileIndex::fileSliceSize)
+ .sum();
+
+ // If the partition value contains InternalRow.empty, we query it as a non-partitioned table.
+ queryAsNonePartitionedTable = partitionFiles.keySet().stream().anyMatch(p -> p.values.length == 0);
+
+ long duration = System.currentTimeMillis() - startTime;
+
+ LOG.info(String.format("Refresh table %s, spent: %d ms", metaClient.getTableConfig().getTableName(), duration));
+ }
+
+ private static long fileSliceSize(FileSlice fileSlice) {
+ long logFileSize = fileSlice.getLogFiles().map(HoodieLogFile::getFileSize)
+ .filter(s -> s > 0)
+ .reduce(0L, Long::sum);
+
+ return fileSlice.getBaseFile().map(BaseFile::getFileLen).orElse(0L) + logFileSize;
+ }
+
+ protected static final class PartitionPath {
+ final String path;
+ final Object[] values;
+
+ public PartitionPath(String path, Object[] values) {
+ this.path = path;
+ this.values = values;
+ }
+
+ Path fullPartitionPath(String basePath) {
+ if (!path.isEmpty()) {
+ return new Path(basePath, path);
+ }
+
+ return new Path(basePath);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof PartitionPath
+ && Objects.equals(path, ((PartitionPath) other).path)
+ && Arrays.equals(values, ((PartitionPath) other).values);
Review comment:
Good point, there's no reason we should be ignoring those, but let me see if this might be breaking compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r801027469
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
Review comment:
Correct, Scala version will be removed along with all deps (kept it for easier comparison)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1031887707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d12ce874c3bfb9861b00449b90ebde0125388b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715)
* 9c7024e7f5e09b48b4da580a4fed0c458a566d99 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1029517702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d12ce874c3bfb9861b00449b90ebde0125388b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r799903283
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ refresh0();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void refresh0() {
Review comment:
Good call
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
Review comment:
Yeah, there are still few cleanups pending (removing the Scala one, renaming this to replace it, etc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1034221722
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814",
"triggerID" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5850",
"triggerID" : "1034220069",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5850)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1028436169
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54e68ec73aa7292556d5f00c90859a776acc109f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1029465605
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54e68ec73aa7292556d5f00c90859a776acc109f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687)
* 7d12ce874c3bfb9861b00449b90ebde0125388b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1019032545
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89266f9e490e4129431d15ac4c7103eb799540e3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1019042151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r800921844
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/Option.java
##########
@@ -123,6 +123,13 @@ public T orElse(T other) {
return val != null ? val : other;
}
+ /**
+ * Returns this {@link Option} if non-empty, and returns the given one otherwise
+ */
+ public Option<T> orElseOption(Option<T> other) {
+ return val != null ? this : other;
+ }
+
Review comment:
Oh, good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1034281138
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814",
"triggerID" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5850",
"triggerID" : "1034220069",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5850)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802081284
##########
File path: packaging/hudi-hadoop-mr-bundle/pom.xml
##########
@@ -67,9 +67,6 @@
<include>org.apache.hudi:hudi-common</include>
<include>org.apache.hudi:hudi-hadoop-mr</include>
- <!-- TODO(HUDI-3239) remove this -->
- <include>org.scala-lang:scala-library</include>
-
Review comment:
Oh, great catch!
Did mindless search and delete somehow bundles didn't bubble up in it. Sorry about that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802144813
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
@yihua : may I know why this is marked as resovled ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1031887707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d12ce874c3bfb9861b00449b90ebde0125388b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715)
* 9c7024e7f5e09b48b4da580a4fed0c458a566d99 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r800923268
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
Review comment:
There's no point in lazy-eval since this class is used eagerly in the flows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802078183
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
Review comment:
Yeah, I see `specifiedQueryInstant: Option[String] = None, shouldIncludePendingCommits: Boolean = false,` defaults in the scala constructor, which can be optional args. But looks like the init without such args are not present anywhere.
```
abstract class HoodieTableFileIndexBase(engineContext: HoodieEngineContext,
metaClient: HoodieTableMetaClient,
configProperties: TypedProperties,
queryType: HoodieTableQueryType,
protected val queryPaths: Seq[Path],
specifiedQueryInstant: Option[String] = None,
shouldIncludePendingCommits: Boolean = false,
@transient fileStatusCache: FileStatusCacheTrait)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1033226075
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c7024e7f5e09b48b4da580a4fed0c458a566d99 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802144296
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
for eg, do we need to add it [here](https://github.com/apache/hudi/blob/973087f385f0e42bd42e2d57008a756e182041ff/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala#L851)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r806419883
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
I don't blame @nsivabalan for the PTSD :) Good to see the paranoia. Thats the first thought that flashed into my mind as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r806419883
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
I don't blame @nsivabalan for the PTSD :) Good to see the paranoia. Thats the first thought that flashed into my mind as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020783178
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f01e128350dd5910ddc51865303df29170317ac0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482)
* 851e61cdb31748501d28d638bc192ccc4955d665 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020783178
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f01e128350dd5910ddc51865303df29170317ac0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482)
* 851e61cdb31748501d28d638bc192ccc4955d665 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020641005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1029467643
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54e68ec73aa7292556d5f00c90859a776acc109f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687)
* 7d12ce874c3bfb9861b00449b90ebde0125388b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1019032545
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89266f9e490e4129431d15ac4c7103eb799540e3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020643261
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89266f9e490e4129431d15ac4c7103eb799540e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429)
* f01e128350dd5910ddc51865303df29170317ac0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1020696601
@alexeykudinkin Thanks for your contribution! Can you figure out why CI failed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r799920946
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
Review comment:
here we should use the naming of `getLatestCompletedInstant` :)
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
Review comment:
Not strictly necessary: I see the scala code has defaults for these two args. Another constructor with these defaults can be added.
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ refresh0();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void refresh0() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.orElseOption(latestInstant.map(HoodieInstant::getTimestamp));
+
+ if (tableType.equals(HoodieTableType.MERGE_ON_READ) && queryType.equals(HoodieTableQueryType.SNAPSHOT)) {
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+ fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
+ .collect(Collectors.toList())
+ )
+ .orElse(Collections.emptyList())
+ )
+ );
+ } else {
+ // TODO re-align with the branch (MOR, snapshot) branch
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ specifiedQueryInstant.map(instant ->
+ fileSystemView.getLatestFileSlicesBeforeOrOn(partitionPath.path, instant, true))
+ .orElse(fileSystemView.getLatestFileSlices(partitionPath.path))
+ .collect(Collectors.toList())
+ )
+ );
+ }
+
+ cachedFileSize = cachedAllInputFileSlices.values().stream()
+ .flatMap(Collection::stream)
+ .mapToLong(JavaBaseHoodieTableFileIndex::fileSliceSize)
+ .sum();
+
+ // If the partition value contains InternalRow.empty, we query it as a non-partitioned table.
+ queryAsNonePartitionedTable = partitionFiles.keySet().stream().anyMatch(p -> p.values.length == 0);
+
+ long duration = System.currentTimeMillis() - startTime;
+
+ LOG.info(String.format("Refresh table %s, spent: %d ms", metaClient.getTableConfig().getTableName(), duration));
+ }
+
+ private static long fileSliceSize(FileSlice fileSlice) {
+ long logFileSize = fileSlice.getLogFiles().map(HoodieLogFile::getFileSize)
+ .filter(s -> s > 0)
+ .reduce(0L, Long::sum);
+
+ return fileSlice.getBaseFile().map(BaseFile::getFileLen).orElse(0L) + logFileSize;
+ }
+
+ protected static final class PartitionPath {
+ final String path;
+ final Object[] values;
+
+ public PartitionPath(String path, Object[] values) {
+ this.path = path;
+ this.values = values;
+ }
+
+ Path fullPartitionPath(String basePath) {
+ if (!path.isEmpty()) {
+ return new Path(basePath, path);
+ }
+
+ return new Path(basePath);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof PartitionPath
+ && Objects.equals(path, ((PartitionPath) other).path)
+ && Arrays.equals(values, ((PartitionPath) other).values);
+ }
+
+ @Override
+ public int hashCode() {
+ return path.hashCode() * 1103 + Arrays.hashCode(values);
+ }
+ }
+
+ protected interface FileStatusCacheTrait {
Review comment:
`Trait` is a scala term. Naming like `FileStatusCache` would be better.
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
Review comment:
The logic is actually the same so this is fine.
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ refresh0();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void refresh0() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.orElseOption(latestInstant.map(HoodieInstant::getTimestamp));
+
+ if (tableType.equals(HoodieTableType.MERGE_ON_READ) && queryType.equals(HoodieTableQueryType.SNAPSHOT)) {
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+ fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
+ .collect(Collectors.toList())
+ )
+ .orElse(Collections.emptyList())
+ )
+ );
+ } else {
+ // TODO re-align with the branch (MOR, snapshot) branch
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ specifiedQueryInstant.map(instant ->
+ fileSystemView.getLatestFileSlicesBeforeOrOn(partitionPath.path, instant, true))
+ .orElse(fileSystemView.getLatestFileSlices(partitionPath.path))
+ .collect(Collectors.toList())
+ )
+ );
+ }
+
+ cachedFileSize = cachedAllInputFileSlices.values().stream()
+ .flatMap(Collection::stream)
+ .mapToLong(JavaBaseHoodieTableFileIndex::fileSliceSize)
+ .sum();
+
+ // If the partition value contains InternalRow.empty, we query it as a non-partitioned table.
+ queryAsNonePartitionedTable = partitionFiles.keySet().stream().anyMatch(p -> p.values.length == 0);
+
+ long duration = System.currentTimeMillis() - startTime;
+
+ LOG.info(String.format("Refresh table %s, spent: %d ms", metaClient.getTableConfig().getTableName(), duration));
+ }
+
+ private static long fileSliceSize(FileSlice fileSlice) {
+ long logFileSize = fileSlice.getLogFiles().map(HoodieLogFile::getFileSize)
+ .filter(s -> s > 0)
+ .reduce(0L, Long::sum);
+
+ return fileSlice.getBaseFile().map(BaseFile::getFileLen).orElse(0L) + logFileSize;
+ }
+
+ protected static final class PartitionPath {
+ final String path;
+ final Object[] values;
+
+ public PartitionPath(String path, Object[] values) {
+ this.path = path;
+ this.values = values;
+ }
+
+ Path fullPartitionPath(String basePath) {
+ if (!path.isEmpty()) {
+ return new Path(basePath, path);
+ }
+
+ return new Path(basePath);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof PartitionPath
+ && Objects.equals(path, ((PartitionPath) other).path)
+ && Arrays.equals(values, ((PartitionPath) other).values);
Review comment:
Is this needed? Previously equality check of values is absent.
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ refresh0();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void refresh0() {
Review comment:
+1
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
Review comment:
Could this be a `List<String>`?
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
Review comment:
As a note, these are previously lazy evaluated in scala. Since they have low overhead, it's okay to initialize them here.
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ refresh0();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void refresh0() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.orElseOption(latestInstant.map(HoodieInstant::getTimestamp));
+
+ if (tableType.equals(HoodieTableType.MERGE_ON_READ) && queryType.equals(HoodieTableQueryType.SNAPSHOT)) {
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+ fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
+ .collect(Collectors.toList())
+ )
+ .orElse(Collections.emptyList())
+ )
+ );
+ } else {
+ // TODO re-align with the branch (MOR, snapshot) branch
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ specifiedQueryInstant.map(instant ->
+ fileSystemView.getLatestFileSlicesBeforeOrOn(partitionPath.path, instant, true))
+ .orElse(fileSystemView.getLatestFileSlices(partitionPath.path))
+ .collect(Collectors.toList())
+ )
+ );
+ }
+
+ cachedFileSize = cachedAllInputFileSlices.values().stream()
+ .flatMap(Collection::stream)
+ .mapToLong(JavaBaseHoodieTableFileIndex::fileSliceSize)
+ .sum();
+
+ // If the partition value contains InternalRow.empty, we query it as a non-partitioned table.
+ queryAsNonePartitionedTable = partitionFiles.keySet().stream().anyMatch(p -> p.values.length == 0);
+
+ long duration = System.currentTimeMillis() - startTime;
+
+ LOG.info(String.format("Refresh table %s, spent: %d ms", metaClient.getTableConfig().getTableName(), duration));
+ }
+
+ private static long fileSliceSize(FileSlice fileSlice) {
+ long logFileSize = fileSlice.getLogFiles().map(HoodieLogFile::getFileSize)
+ .filter(s -> s > 0)
+ .reduce(0L, Long::sum);
+
+ return fileSlice.getBaseFile().map(BaseFile::getFileLen).orElse(0L) + logFileSize;
Review comment:
nit: no need for variable `logFileSize`, to directly `return fileSlice.getBaseFile()... + fileSlice.getLogFiles()...`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1031884934
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d12ce874c3bfb9861b00449b90ebde0125388b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715)
* 9c7024e7f5e09b48b4da580a4fed0c458a566d99 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r799861412
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
Review comment:
note to self. In master, we have if else for partitioned or non partitioned dataset. verify if its taken care of else where in this file (where cachedAllInputFileSlices are populated)
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
Review comment:
is this replacement for HoodieTableFileIndexBase? Can we avoid "Java" as suffix in the class name. We have java engine and might be confusing. May be we can just keep it as "BaseHoodieTableFileIndex"?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
this might be break things for users. I mean, this is what existing users might be using in their pipelines to read hudi tables. Let's be cautious in these renames.
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ refresh0();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void refresh0() {
Review comment:
can we fix the naming. refreshInternal or something. refresh0 is not a good naming.
##########
File path: hudi-common/src/main/scala/org/apache/hudi/BaseHoodieTableFileIndex.scala
##########
@@ -51,19 +51,14 @@ import scala.language.implicitConversions
* @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
* @param fileStatusCache transient cache of fetched [[FileStatus]]es
*/
-abstract class HoodieTableFileIndexBase(engineContext: HoodieEngineContext,
+abstract class BaseHoodieTableFileIndex(engineContext: HoodieEngineContext,
Review comment:
there is a large overlap between this and JavaBaseHoodieTableFileIndex right. do we plan to dedup in follow up patch ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r803112903
##########
File path: hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class BaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCache fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCache fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ doRefresh();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> getLatestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ doRefresh();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void doRefresh() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.or(() -> latestInstant.map(HoodieInstant::getTimestamp));
Review comment:
No, it will just produce empty Option
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1028372962
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 851e61cdb31748501d28d638bc192ccc4955d665 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494)
* 54e68ec73aa7292556d5f00c90859a776acc109f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1028370459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 851e61cdb31748501d28d638bc192ccc4955d665 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494)
* 54e68ec73aa7292556d5f00c90859a776acc109f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802251345
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
ok, my bad totally. I got confused with DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL and other variables. we are good here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802100592
##########
File path: hudi-common/pom.xml
##########
@@ -38,18 +38,6 @@
<pluginManagement>
<plugins>
- <plugin>
- <groupId>net.alchim31.maven</groupId>
- <artifactId>scala-maven-plugin</artifactId>
- <version>${scala-maven-plugin.version}</version>
- <configuration>
- <args>
- <arg>-nobootcp</arg>
- <arg>-target:jvm-1.8</arg>
- </args>
- <checkMultipleScalaVersions>false</checkMultipleScalaVersions>
- </configuration>
- </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
Review comment:
Should the `<pluginManagement>` be removed altogether? I see maven-compiler-plugin is added before mainly for compiling both Java and Scala code.
##########
File path: hudi-common/pom.xml
##########
@@ -58,27 +46,6 @@
</pluginManagement>
<plugins>
- <plugin>
- <groupId>net.alchim31.maven</groupId>
- <artifactId>scala-maven-plugin</artifactId>
- <executions>
- <execution>
- <id>scala-compile-first</id>
- <phase>process-resources</phase>
- <goals>
- <goal>add-source</goal>
- <goal>compile</goal>
- </goals>
- </execution>
- <execution>
- <id>scala-test-compile</id>
- <phase>process-test-resources</phase>
- <goals>
- <goal>testCompile</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
Review comment:
same here for this plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1033228002
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c7024e7f5e09b48b4da580a4fed0c458a566d99 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777)
* dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1033265778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814",
"triggerID" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c7024e7f5e09b48b4da580a4fed0c458a566d99 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777)
* dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802073658
##########
File path: hudi-common/src/main/java/org/apache/hudi/JavaBaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class JavaBaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(JavaBaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCacheTrait fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public JavaBaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCacheTrait fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ refresh0();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> latestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ refresh0();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void refresh0() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.orElseOption(latestInstant.map(HoodieInstant::getTimestamp));
+
+ if (tableType.equals(HoodieTableType.MERGE_ON_READ) && queryType.equals(HoodieTableQueryType.SNAPSHOT)) {
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+ fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path, queryInstant.get())
+ .collect(Collectors.toList())
+ )
+ .orElse(Collections.emptyList())
+ )
+ );
+ } else {
+ // TODO re-align with the branch (MOR, snapshot) branch
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ specifiedQueryInstant.map(instant ->
+ fileSystemView.getLatestFileSlicesBeforeOrOn(partitionPath.path, instant, true))
+ .orElse(fileSystemView.getLatestFileSlices(partitionPath.path))
+ .collect(Collectors.toList())
+ )
+ );
+ }
+
+ cachedFileSize = cachedAllInputFileSlices.values().stream()
+ .flatMap(Collection::stream)
+ .mapToLong(JavaBaseHoodieTableFileIndex::fileSliceSize)
+ .sum();
+
+ // If the partition value contains InternalRow.empty, we query it as a non-partitioned table.
+ queryAsNonePartitionedTable = partitionFiles.keySet().stream().anyMatch(p -> p.values.length == 0);
+
+ long duration = System.currentTimeMillis() - startTime;
+
+ LOG.info(String.format("Refresh table %s, spent: %d ms", metaClient.getTableConfig().getTableName(), duration));
+ }
+
+ private static long fileSliceSize(FileSlice fileSlice) {
+ long logFileSize = fileSlice.getLogFiles().map(HoodieLogFile::getFileSize)
+ .filter(s -> s > 0)
+ .reduce(0L, Long::sum);
+
+ return fileSlice.getBaseFile().map(BaseFile::getFileLen).orElse(0L) + logFileSize;
+ }
+
+ protected static final class PartitionPath {
+ final String path;
+ final Object[] values;
+
+ public PartitionPath(String path, Object[] values) {
+ this.path = path;
+ this.values = values;
+ }
+
+ Path fullPartitionPath(String basePath) {
+ if (!path.isEmpty()) {
+ return new Path(basePath, path);
+ }
+
+ return new Path(basePath);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof PartitionPath
+ && Objects.equals(path, ((PartitionPath) other).path)
+ && Arrays.equals(values, ((PartitionPath) other).values);
Review comment:
What's your take here? If not sure, it's better to maintain the same behavior as before to be safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#discussion_r802162914
##########
File path: hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class BaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCache fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCache fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ doRefresh();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> getLatestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ doRefresh();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
+
+ if (pathToFetch.isEmpty()) {
+ fetchedPartitionToFiles = Collections.emptyMap();
+ } else {
+ Map<String, PartitionPath> fullPartitionPathsMapToFetch = pathToFetch.stream()
+ .collect(Collectors.toMap(
+ partitionPath -> partitionPath.fullPartitionPath(basePath).toString(),
+ Function.identity())
+ );
+
+ fetchedPartitionToFiles =
+ FSUtils.getFilesInPartitions(
+ engineContext,
+ metadataConfig,
+ basePath,
+ fullPartitionPathsMapToFetch.keySet().toArray(new String[0]),
+ fileSystemStorageConfig.getSpillableDir())
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> fullPartitionPathsMapToFetch.get(e.getKey()), e -> e.getValue()));
+
+ }
+
+ // Update the fileStatusCache
+ fetchedPartitionToFiles.forEach((partitionPath, filesInPartition) -> {
+ fileStatusCache.put(partitionPath.fullPartitionPath(basePath), filesInPartition);
+ });
+
+ return CollectionUtils.combine(cachedPartitionToFiles, fetchedPartitionToFiles);
+ }
+
+ private void doRefresh() {
+ long startTime = System.currentTimeMillis();
+
+ Map<PartitionPath, FileStatus[]> partitionFiles = loadPartitionPathFiles();
+ FileStatus[] allFiles = partitionFiles.values().stream().flatMap(Arrays::stream).toArray(FileStatus[]::new);
+
+ metaClient.reloadActiveTimeline();
+
+ HoodieTimeline activeTimeline = getActiveTimeline();
+ Option<HoodieInstant> latestInstant = activeTimeline.lastInstant();
+
+ // TODO we can optimize the flow by:
+ // - First fetch list of files from instants of interest
+ // - Load FileStatus's
+ fileSystemView = new HoodieTableFileSystemView(metaClient, activeTimeline, allFiles);
+
+ Option<String> queryInstant =
+ specifiedQueryInstant.or(() -> latestInstant.map(HoodieInstant::getTimestamp));
Review comment:
Not sure i follow: do you mean what happens here in this expression or in the flow in general?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieTableQueryType.java
##########
@@ -30,7 +30,7 @@
* </ol>
*/
public enum HoodieTableQueryType {
- QUERY_TYPE_SNAPSHOT,
- QUERY_TYPE_INCREMENTAL,
- QUERY_TYPE_READ_OPTIMIZED
+ SNAPSHOT,
Review comment:
Not sure i understand where the disconnect is: there's no old variables -- this enum has been introduced in one of my preceding PRs in this same stack. As such, nobody could possibly be using it unless they depend on snapshot artifacts, which i deem unlikely.
Moreover, internal components that are not annotated as _public API_ are implicitly considered private components and do not the same compatibility guarantees attached as do the Public APIs, and therefore should not be dependent upon as these could change w/o notice.
##########
File path: hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieTableQueryType;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Common (engine-agnostic) File Index implementation enabling individual query engines to
+ * list Hudi Table contents based on the
+ *
+ * <ul>
+ * <li>Table type (MOR, COW)</li>
+ * <li>Query type (snapshot, read_optimized, incremental)</li>
+ * <li>Query instant/range</li>
+ * </ul>
+ */
+public abstract class BaseHoodieTableFileIndex {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieTableFileIndex.class);
+
+ private final String[] partitionColumns;
+
+ private final FileSystemViewStorageConfig fileSystemStorageConfig;
+ private final HoodieMetadataConfig metadataConfig;
+
+ private final HoodieTableQueryType queryType;
+ private final Option<String> specifiedQueryInstant;
+ protected final List<Path> queryPaths;
+
+ private final boolean shouldIncludePendingCommits;
+
+ private final HoodieTableType tableType;
+ protected final String basePath;
+
+ private final HoodieTableMetaClient metaClient;
+ private final HoodieEngineContext engineContext;
+
+ private final transient FileStatusCache fileStatusCache;
+
+ protected transient volatile long cachedFileSize = 0L;
+ protected transient volatile Map<PartitionPath, List<FileSlice>> cachedAllInputFileSlices;
+
+ protected volatile boolean queryAsNonePartitionedTable = false;
+
+ private transient volatile HoodieTableFileSystemView fileSystemView = null;
+
+ /**
+ * @param engineContext Hudi engine-specific context
+ * @param metaClient Hudi table's meta-client
+ * @param configProperties unifying configuration (in the form of generic properties)
+ * @param queryType target query type
+ * @param queryPaths target DFS paths being queried
+ * @param specifiedQueryInstant instant as of which table is being queried
+ * @param shouldIncludePendingCommits flags whether file-index should exclude any pending operations
+ * @param fileStatusCache transient cache of fetched [[FileStatus]]es
+ */
+ public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
+ HoodieTableMetaClient metaClient,
+ TypedProperties configProperties,
+ HoodieTableQueryType queryType,
+ List<Path> queryPaths,
+ Option<String> specifiedQueryInstant,
+ boolean shouldIncludePendingCommits,
+ FileStatusCache fileStatusCache) {
+ this.partitionColumns = metaClient.getTableConfig().getPartitionFields()
+ .orElse(new String[0]);
+
+ this.fileSystemStorageConfig = FileSystemViewStorageConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+ this.metadataConfig = HoodieMetadataConfig.newBuilder()
+ .fromProperties(configProperties)
+ .build();
+
+ this.queryType = queryType;
+ this.queryPaths = queryPaths;
+ this.specifiedQueryInstant = specifiedQueryInstant;
+ this.shouldIncludePendingCommits = shouldIncludePendingCommits;
+
+ this.tableType = metaClient.getTableType();
+ this.basePath = metaClient.getBasePath();
+
+ this.metaClient = metaClient;
+ this.engineContext = engineContext;
+ this.fileStatusCache = fileStatusCache;
+
+ doRefresh();
+ }
+
+ protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns, String partitionPath);
+
+ /**
+ * Returns latest completed instant as seen by this instance of the file-index
+ */
+ public Option<HoodieInstant> getLatestCompletedInstant() {
+ return getActiveTimeline().filterCompletedInstants().lastInstant();
+ }
+
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @return mapping from string partition paths to its base/log files
+ */
+ public Map<String, List<FileSlice>> listFileSlices() {
+ return cachedAllInputFileSlices.entrySet()
+ .stream()
+ .collect(Collectors.toMap(e -> e.getKey().path, Map.Entry::getValue));
+ }
+
+ protected List<PartitionPath> getAllQueryPartitionPaths() {
+ List<String> queryRelativePartitionPaths = queryPaths.stream()
+ .map(path -> FSUtils.getRelativePartitionPath(new Path(basePath), path))
+ .collect(Collectors.toList());
+
+ // Load all the partition path from the basePath, and filter by the query partition path.
+ // TODO load files from the queryRelativePartitionPaths directly.
+ List<String> matchedPartitionPaths = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath)
+ .stream()
+ .filter(path -> queryRelativePartitionPaths.stream().anyMatch(path::startsWith))
+ .collect(Collectors.toList());
+
+ // Convert partition's path into partition descriptor
+ return matchedPartitionPaths.stream()
+ .map(partitionPath -> {
+ Object[] partitionColumnValues = parsePartitionColumnValues(partitionColumns, partitionPath);
+ return new PartitionPath(partitionPath, partitionColumnValues);
+ })
+ .collect(Collectors.toList());
+ }
+
+ protected void refresh() {
+ fileStatusCache.invalidate();
+ doRefresh();
+ }
+
+ protected HoodieTimeline getActiveTimeline() {
+ // NOTE: We have to use commits and compactions timeline, to make sure that we're properly
+ // handling the following case: when records are inserted into the new log-file w/in the file-group
+ // that is under the pending compaction process, new log-file will bear the compaction's instant (on the
+ // timeline) in its name, as opposed to the base-file's commit instant. To make sure we're not filtering
+ // such log-file we have to _always_ include pending compaction instants into consideration
+ // TODO(HUDI-3302) re-evaluate whether we should not filter any commits in here
+ HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline();
+ if (shouldIncludePendingCommits) {
+ return timeline;
+ } else {
+ return timeline.filterCompletedAndCompactionInstants();
+ }
+ }
+
+ /**
+ * Load all partition paths and it's files under the query table path.
+ */
+ private Map<PartitionPath, FileStatus[]> loadPartitionPathFiles() {
+ // List files in all partition paths
+ List<PartitionPath> pathToFetch = new ArrayList<>();
+ Map<PartitionPath, FileStatus[]> cachedPartitionToFiles = new HashMap<>();
+
+ // Fetch from the FileStatusCache
+ List<PartitionPath> partitionPaths = getAllQueryPartitionPaths();
+ partitionPaths.forEach(partitionPath -> {
+ Option<FileStatus[]> filesInPartition = fileStatusCache.get(partitionPath.fullPartitionPath(basePath));
+ if (filesInPartition.isPresent()) {
+ cachedPartitionToFiles.put(partitionPath, filesInPartition.get());
+ } else {
+ pathToFetch.add(partitionPath);
+ }
+ });
+
+ Map<PartitionPath, FileStatus[]> fetchedPartitionToFiles;
Review comment:
That was intentional so that the variable stays immutable (and is reflected as such by IDE)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1034221722
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814",
"triggerID" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5850",
"triggerID" : "1034220069",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5850)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1033327239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
}, {
"hash" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5715",
"triggerID" : "7d12ce874c3bfb9861b00449b90ebde0125388b7",
"triggerType" : "PUSH"
}, {
"hash" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5777",
"triggerID" : "9c7024e7f5e09b48b4da580a4fed0c458a566d99",
"triggerType" : "PUSH"
}, {
"hash" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814",
"triggerID" : "dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd2daec9f0bead9b6e7e896e7d1faf1f9a7959e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5814)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1019042151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89266f9e490e4129431d15ac4c7103eb799540e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1019042151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89266f9e490e4129431d15ac4c7103eb799540e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4669: [HUDI-3239][Stacked on 4667] Convert `BaseHoodieTableFileIndex` to Java
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4669:
URL: https://github.com/apache/hudi/pull/4669#issuecomment-1028436169
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5429",
"triggerID" : "89266f9e490e4129431d15ac4c7103eb799540e3",
"triggerType" : "PUSH"
}, {
"hash" : "f01e128350dd5910ddc51865303df29170317ac0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5482",
"triggerID" : "f01e128350dd5910ddc51865303df29170317ac0",
"triggerType" : "PUSH"
}, {
"hash" : "851e61cdb31748501d28d638bc192ccc4955d665",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5494",
"triggerID" : "851e61cdb31748501d28d638bc192ccc4955d665",
"triggerType" : "PUSH"
}, {
"hash" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687",
"triggerID" : "54e68ec73aa7292556d5f00c90859a776acc109f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54e68ec73aa7292556d5f00c90859a776acc109f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5687)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org