You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/17 07:44:58 UTC
[GitHub] [hudi] boneanxs opened a new pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
boneanxs opened a new pull request #4014:
URL: https://github.com/apache/hudi/pull/4014
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Cache BaseDir instead of folder to improve performance if `HudiTableNotFound `exception thrown.
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
This pull request is already covered by existing tests, such as *(please describe tests)*.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-977692550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on a change in pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
boneanxs commented on a change in pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#discussion_r755791546
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -173,6 +173,13 @@ public boolean accept(Path path) {
}
if (baseDir != null) {
+ // Check whether baseDir in nonHoodiePathCache
+ if (nonHoodiePathCache.contains(baseDir.toString())) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Accepting non-hoodie path from cache: " + path);
+ }
+ return true;
+ }
Review comment:
There already have the tests covered multi files have the same baseDir(`testPartitionPathsAsNonHoodiePaths`), So I just add the size check, is it enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-977635366
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424)
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-977637829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424)
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-971356334
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-971320893
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
boneanxs commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-971321742
@codope, @nsivabalan Can you help to approve running workflows?
relate to this discussion: [Maybe should cache baseDir in nonHoodiePathCache in HoodieROTablePathFilter](https://github.com/apache/hudi/issues/3856)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-983611140
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
}, {
"hash" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677)
* ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-983758117
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
}, {
"hash" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918",
"triggerID" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"triggerType" : "PUSH"
}, {
"hash" : "d3de6ffe7f4d60ff102be081b18eaa0311854d2f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d3de6ffe7f4d60ff102be081b18eaa0311854d2f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918)
* d3de6ffe7f4d60ff102be081b18eaa0311854d2f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-977635366
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424)
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-977633174
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424)
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-977637829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424)
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-983729264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
}, {
"hash" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918",
"triggerID" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-983663665
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
}, {
"hash" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918",
"triggerID" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677)
* ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-983611140
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
}, {
"hash" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677)
* ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-983758117
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
}, {
"hash" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918",
"triggerID" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"triggerType" : "PUSH"
}, {
"hash" : "d3de6ffe7f4d60ff102be081b18eaa0311854d2f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d3de6ffe7f4d60ff102be081b18eaa0311854d2f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918)
* d3de6ffe7f4d60ff102be081b18eaa0311854d2f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
boneanxs commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-975041971
@codope, @nsivabalan, gentle ping... Could you pls take a look, also cc @vinothchandar , maybe you can give us more inputs as you implemented this PathFilter :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#discussion_r754183807
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -173,6 +173,13 @@ public boolean accept(Path path) {
}
if (baseDir != null) {
+ // Check whether baseDir in nonHoodiePathCache
+ if (nonHoodiePathCache.contains(baseDir.toString())) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Accepting non-hoodie path from cache: " + path);
+ }
+ return true;
+ }
Review comment:
Can you add a test case to cover this scenario? You can check existing tests in `TestHoodieROTablePathFilter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-983663665
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
}, {
"hash" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918",
"triggerID" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677)
* ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on a change in pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
boneanxs commented on a change in pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#discussion_r760148712
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -173,6 +173,13 @@ public boolean accept(Path path) {
}
if (baseDir != null) {
+ // Check whether baseDir in nonHoodiePathCache
+ if (nonHoodiePathCache.contains(baseDir.toString())) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Accepting non-hoodie path from cache: " + path);
+ }
+ return true;
+ }
Review comment:
Oh, I finally get your point, `.hoodie` dir get accepted or not is checked by test `testHoodiePaths`
```java
assertFalse(pathFilter.accept(new Path("file:///" + basePath + "/" + HoodieTableMetaClient.METAFOLDER_NAME + "/")));
assertFalse(pathFilter.accept(new Path("file:///" + basePath + "/" + HoodieTableMetaClient.METAFOLDER_NAME)));
```
Also add files under `.hoodie` check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
boneanxs commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-989842423
Thanks a lot @codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
boneanxs commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-979595271
@codope Pls take a look:D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-971322466
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on a change in pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
boneanxs commented on a change in pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#discussion_r760151618
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -231,6 +240,12 @@ public boolean accept(Path path) {
}
}
+ @VisibleForTesting
Review comment:
As CheckStyle doesn't encourage to import `com.google.common.annotations.VisibleForTesting` annotation. I simply follow `metaClientCache` to make `nonHoodiePathCache` access specifier is `friendly` for test purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-971356334
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-971320893
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
codope merged pull request #4014:
URL: https://github.com/apache/hudi/pull/4014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-977692550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
boneanxs commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-977610252
> @boneanxs Thanks for this contribution. I have a question. What happens for hidden files and directories? Will it not return true for them if we only check the basePath?
If the baseDir of hidden files/directories is in nonHoodieCache, it will return true directly, otherwise will follow the old code path to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-977633174
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424)
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
codope commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-979700587
> > @boneanxs Thanks for this contribution. I have a question. What happens for hidden files and directories? Will it not return true for them if we only check the basePath?
>
> If the baseDir of hidden files/directories is in nonHoodieCache, it will return true directly, otherwise will follow the old code path to check.
But then the hoodie timeline files under `.hoodie` directory would be accepted as well even though that's not a valid data partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-983729264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
}, {
"hash" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918",
"triggerID" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-971322466
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa4b108f2ad33de2a0f16794a2aad15133edec5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
boneanxs commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-986363754
Hi @codope, Could you pls review this again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
codope commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-986748604
> Hi @codope, Could you pls review this again?
Overall looks good. But, need to think of a few cases. Will get back to this by wednesday.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#issuecomment-983760794
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3424",
"triggerID" : "5fa4b108f2ad33de2a0f16794a2aad15133edec5",
"triggerType" : "PUSH"
}, {
"hash" : "476cd38123d0c6303b844767ab2e48219a149320",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "476cd38123d0c6303b844767ab2e48219a149320",
"triggerType" : "PUSH"
}, {
"hash" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3677",
"triggerID" : "a752e0a1c2367e9e1cfe54ecfd40e7b02ab64b5e",
"triggerType" : "PUSH"
}, {
"hash" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3918",
"triggerID" : "ed0d854d9cc1be0b2e0c52f8fe3184cbacb1aaa6",
"triggerType" : "PUSH"
}, {
"hash" : "d3de6ffe7f4d60ff102be081b18eaa0311854d2f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3916",
"triggerID" : "d3de6ffe7f4d60ff102be081b18eaa0311854d2f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 476cd38123d0c6303b844767ab2e48219a149320 UNKNOWN
* d3de6ffe7f4d60ff102be081b18eaa0311854d2f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3916)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on a change in pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
boneanxs commented on a change in pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#discussion_r755766928
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -173,6 +173,13 @@ public boolean accept(Path path) {
}
if (baseDir != null) {
+ // Check whether baseDir in nonHoodiePathCache
+ if (nonHoodiePathCache.contains(baseDir.toString())) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Accepting non-hoodie path from cache: " + path);
+ }
+ return true;
+ }
Review comment:
Sure, will add it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4014: [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4014:
URL: https://github.com/apache/hudi/pull/4014#discussion_r757232858
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -211,9 +219,10 @@ public boolean accept(Path path) {
} catch (TableNotFoundException e) {
// Non-hoodie path, accept it.
if (LOG.isDebugEnabled()) {
- LOG.debug(String.format("(1) Caching non-hoodie path under %s \n", folder.toString()));
+ LOG.debug(String.format("(1) Caching non-hoodie path under %s and %s \n", folder.toString(), baseDir.toString()));
Review comment:
`path under %s and %s` might confuse users. Maybe, `path under %s with basePath %s` would be more clear
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -173,6 +173,13 @@ public boolean accept(Path path) {
}
if (baseDir != null) {
+ // Check whether baseDir in nonHoodiePathCache
+ if (nonHoodiePathCache.contains(baseDir.toString())) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Accepting non-hoodie path from cache: " + path);
+ }
+ return true;
+ }
Review comment:
We should check this for hoodie paths as well i.e. see if files under `.hoodie` dir get accepted or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org