You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/02/24 12:54:20 UTC
[GitHub] [hudi] huberylee opened a new pull request #4901: [WIP][HUDI-3445] Clustering Command Based on Call Procedure Command for Spark SQL
huberylee opened a new pull request #4901:
URL: https://github.com/apache/hudi/pull/4901
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*Supporting clustering command for SparkSQL*
## Brief change log
- *Add RunClusteringProcedure and ShowClusteringProcedure*
- *Add test case in TestCallProcedure*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054999877
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410)
* 7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1058120736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422",
"triggerID" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"triggerType" : "PUSH"
}, {
"hash" : "95158068499ec54bf551d9a03ca315e3fa9fb45d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6512",
"triggerID" : "95158068499ec54bf551d9a03ca315e3fa9fb45d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422)
* 95158068499ec54bf551d9a03ca315e3fa9fb45d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6512)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054487071
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a25bc26caa76f22f6f0ca91305906b02d50816f6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307)
* fac4277d943e70f8e0375b386bd197b5f4e88c0c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1049907712
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afde89d0dfa27c145a804d54acfb30b3b21ebc9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1051567205
> @XuQianJin-Stars do you have some time to review this pr, thanks, may be we can add this to [HUDI-3161](https://issues.apache.org/jira/browse/HUDI-3161)
well, Let me review this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [WIP][HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1049907712
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afde89d0dfa27c145a804d54acfb30b3b21ebc9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1050755055
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afde89d0dfa27c145a804d54acfb30b3b21ebc9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273)
* a25bc26caa76f22f6f0ca91305906b02d50816f6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1051549434
@XuQianJin-Stars do you have some time to review this pr, thanks, may be we can add this to HUDI-3161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1055117112
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422",
"triggerID" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054164686
hi @huberylee pls fixed the CI build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1058117823
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422",
"triggerID" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"triggerType" : "PUSH"
}, {
"hash" : "95158068499ec54bf551d9a03ca315e3fa9fb45d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "95158068499ec54bf551d9a03ca315e3fa9fb45d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422)
* 95158068499ec54bf551d9a03ca315e3fa9fb45d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1058062906
This PR looks good overall. Thanks @huberylee
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] huberylee commented on a change in pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
huberylee commented on a change in pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#discussion_r819269475
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieCommonUtils.scala
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hadoop.fs.Path
+import org.apache.hudi.BaseHoodieTableFileIndex.PartitionPath
+import org.apache.hudi.client.SparkRDDWriteClient
+import org.apache.hudi.common.config.{HoodieMetadataConfig, TypedProperties}
+import org.apache.hudi.common.table.{HoodieTableMetaClient, TableSchemaResolver}
+import org.apache.hudi.keygen.{TimestampBasedAvroKeyGenerator, TimestampBasedKeyGenerator}
+import org.apache.spark.api.java.JavaSparkContext
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, BoundReference, Expression, InterpretedPredicate}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LocalRelation}
+import org.apache.spark.sql.catalyst.util.DateTimeUtils
+import org.apache.spark.sql.catalyst.{InternalRow, expressions}
+import org.apache.spark.sql.execution.datasources.SparkParsePartitionUtil
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.withSparkConf
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{StringType, StructField, StructType}
+import org.apache.spark.unsafe.types.UTF8String
+
+import scala.collection.JavaConverters.mapAsJavaMapConverter
+import scala.collection.immutable.Map
+
+object HoodieCommonUtils extends Logging {
+
+ def createHoodieClientFromPath(sparkSession: SparkSession, basePath: String,
+ conf: Map[String, String]): SparkRDDWriteClient[_] = {
+ val metaClient = HoodieTableMetaClient.builder().setBasePath(basePath)
+ .setConf(sparkSession.sessionState.newHadoopConf()).build()
+ val schemaUtil = new TableSchemaResolver(metaClient)
+ val schemaStr = schemaUtil.getTableAvroSchemaWithoutMetadataFields.toString
+ val finalParameters = HoodieWriterUtils.parametersWithWriteDefaults(
+ withSparkConf(sparkSession, Map.empty)(
+ conf + (DataSourceWriteOptions.TABLE_TYPE.key() -> metaClient.getTableType.name()))
+ )
+
+ val jsc = new JavaSparkContext(sparkSession.sparkContext)
+ DataSourceUtils.createHoodieClient(jsc, schemaStr, basePath,
+ metaClient.getTableConfig.getTableName, finalParameters.asJava)
+ }
+
+ def getPartitionSchemaFromProperty(metaClient: HoodieTableMetaClient,
+ schemaSpec: Option[StructType]): StructType = {
+ val schema = schemaSpec.getOrElse({
+ val schemaUtil = new TableSchemaResolver(metaClient)
+ AvroConversionUtils.convertAvroSchemaToStructType(schemaUtil.getTableAvroSchema)
+ })
+
+ val tableConfig = metaClient.getTableConfig
+ val partitionColumns = tableConfig.getPartitionFields
+ val nameFieldMap = generateFieldMap(schema)
+
+ if (partitionColumns.isPresent) {
+ // Note that key generator class name could be null
+ val keyGeneratorClassName = tableConfig.getKeyGeneratorClassName
+ if (classOf[TimestampBasedKeyGenerator].getName.equalsIgnoreCase(keyGeneratorClassName)
+ || classOf[TimestampBasedAvroKeyGenerator].getName.equalsIgnoreCase(keyGeneratorClassName)) {
+ val partitionFields = partitionColumns.get().map(column => StructField(column, StringType))
+ StructType(partitionFields)
+ } else {
+ val partitionFields = partitionColumns.get().map(column =>
+ nameFieldMap.getOrElse(column, throw new IllegalArgumentException(s"Cannot find column: '" +
+ s"$column' in the schema[${schema.fields.mkString(",")}]")))
+ StructType(partitionFields)
+ }
+ } else {
+ // If the partition columns have not stored in hoodie.properties(the table that was
+ // created earlier), we trait it as a non-partitioned table.
+ logWarning("No partition columns available from hoodie.properties." +
+ " Partition pruning will not work")
+ new StructType()
+ }
+ }
+
+ /**
+ * This method unravels [[StructType]] into a [[Map]] of pairs of dot-path notation with corresponding
+ * [[StructField]] object for every field of the provided [[StructType]], recursively.
+ *
+ * For example, following struct
+ * <pre>
Review comment:
Sorry, I'll reformat this comment in https://github.com/apache/hudi/pull/4945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1050813470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a25bc26caa76f22f6f0ca91305906b02d50816f6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1055001727
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422",
"triggerID" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410)
* 7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1058117823
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422",
"triggerID" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"triggerType" : "PUSH"
}, {
"hash" : "95158068499ec54bf551d9a03ca315e3fa9fb45d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "95158068499ec54bf551d9a03ca315e3fa9fb45d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422)
* 95158068499ec54bf551d9a03ca315e3fa9fb45d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] huberylee commented on a change in pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
huberylee commented on a change in pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#discussion_r818688760
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/strategy/PartitionAwareClusteringPlanStrategy.java
##########
@@ -113,6 +114,15 @@ public PartitionAwareClusteringPlanStrategy(HoodieTable table, HoodieEngineConte
.build());
}
+ public List<String> getMatchedPartitions(HoodieWriteConfig config, List<String> partitionPaths) {
+ String partitionSelected = config.getClusteringPartitionSelected();
+ if (!StringUtils.isNullOrEmpty(partitionSelected)) {
+ return Arrays.asList(partitionSelected.split(","));
Review comment:
Yes, the filtered partition is separated by comma when pruning partition, see the end of `org.apache.spark.sql.hudi.command.procedures.RunClusteringProcedure#prunePartition` for more detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] huberylee commented on a change in pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
huberylee commented on a change in pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#discussion_r818703334
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/RunClusteringProcedure.scala
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.BaseHoodieTableFileIndex.PartitionPath
+import org.apache.hudi.client.common.HoodieSparkEngineContext
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig
+import org.apache.hudi.common.table.{HoodieTableMetaClient, TableSchemaResolver}
+import org.apache.hudi.common.util.{ClusteringUtils, Option => HOption}
+import org.apache.hudi.config.HoodieClusteringConfig
+import org.apache.hudi.exception.HoodieClusteringException
+import org.apache.hudi.metadata.HoodieTableMetadata
+import org.apache.hudi.{HoodieCommonUtils, SparkAdapterSupport}
+import org.apache.spark.api.java.JavaSparkContext
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.types._
+
+import java.util.Properties
+import java.util.function.Supplier
+import scala.collection.JavaConverters
+import scala.collection.JavaConverters._
+
+class RunClusteringProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+ /**
+ * OPTIMIZE table_name|table_path [WHERE predicate]
+ * [ORDER BY (col_name1 [, ...] ) ]
+ */
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType, None),
Review comment:
Yes, no matter in `RunClusteringProcedure` or `ShowClusteringProcedure`, table and path are optional, and it's ok to calling those procedures by table or path. What should be noticed is that one of them must be set in one specific call, and this guaranteed by `org.apache.spark.sql.hudi.command.procedures.BaseProcedure#getBasePath`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1051549041
@XuQianJin-Stars do you have some time to review this pr, thanks, may be we can add this to HUDI-3161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1055001727
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422",
"triggerID" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410)
* 7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054929568
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fac4277d943e70f8e0375b386bd197b5f4e88c0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399)
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a change in pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on a change in pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#discussion_r818691924
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/strategy/PartitionAwareClusteringPlanStrategy.java
##########
@@ -113,6 +114,15 @@ public PartitionAwareClusteringPlanStrategy(HoodieTable table, HoodieEngineConte
.build());
}
+ public List<String> getMatchedPartitions(HoodieWriteConfig config, List<String> partitionPaths) {
+ String partitionSelected = config.getClusteringPartitionSelected();
+ if (!StringUtils.isNullOrEmpty(partitionSelected)) {
+ return Arrays.asList(partitionSelected.split(","));
Review comment:
> Yes, the filtered partition is separated by comma when pruning partition, see the end of `org.apache.spark.sql.hudi.command.procedures.RunClusteringProcedure#prunePartition` for more detail.
Change to a constant or set writer's option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1058227335
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422",
"triggerID" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"triggerType" : "PUSH"
}, {
"hash" : "95158068499ec54bf551d9a03ca315e3fa9fb45d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6512",
"triggerID" : "95158068499ec54bf551d9a03ca315e3fa9fb45d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95158068499ec54bf551d9a03ca315e3fa9fb45d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6512)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054483868
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a25bc26caa76f22f6f0ca91305906b02d50816f6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307)
* fac4277d943e70f8e0375b386bd197b5f4e88c0c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1050757327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afde89d0dfa27c145a804d54acfb30b3b21ebc9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273)
* a25bc26caa76f22f6f0ca91305906b02d50816f6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [WIP][HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1049836222
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afde89d0dfa27c145a804d54acfb30b3b21ebc9f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [WIP][HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1049833521
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afde89d0dfa27c145a804d54acfb30b3b21ebc9f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054547429
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fac4277d943e70f8e0375b386bd197b5f4e88c0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054927965
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fac4277d943e70f8e0375b386bd197b5f4e88c0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399)
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054483868
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a25bc26caa76f22f6f0ca91305906b02d50816f6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307)
* fac4277d943e70f8e0375b386bd197b5f4e88c0c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [WIP][HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1049833521
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afde89d0dfa27c145a804d54acfb30b3b21ebc9f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1050757327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afde89d0dfa27c145a804d54acfb30b3b21ebc9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273)
* a25bc26caa76f22f6f0ca91305906b02d50816f6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] huberylee commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
huberylee commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054427848
@xiarixiaoyao @XuQianJin-Stars Sorry for the late reply, I will fix it wright now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#discussion_r819258970
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieCommonUtils.scala
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hadoop.fs.Path
+import org.apache.hudi.BaseHoodieTableFileIndex.PartitionPath
+import org.apache.hudi.client.SparkRDDWriteClient
+import org.apache.hudi.common.config.{HoodieMetadataConfig, TypedProperties}
+import org.apache.hudi.common.table.{HoodieTableMetaClient, TableSchemaResolver}
+import org.apache.hudi.keygen.{TimestampBasedAvroKeyGenerator, TimestampBasedKeyGenerator}
+import org.apache.spark.api.java.JavaSparkContext
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, BoundReference, Expression, InterpretedPredicate}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LocalRelation}
+import org.apache.spark.sql.catalyst.util.DateTimeUtils
+import org.apache.spark.sql.catalyst.{InternalRow, expressions}
+import org.apache.spark.sql.execution.datasources.SparkParsePartitionUtil
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.withSparkConf
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{StringType, StructField, StructType}
+import org.apache.spark.unsafe.types.UTF8String
+
+import scala.collection.JavaConverters.mapAsJavaMapConverter
+import scala.collection.immutable.Map
+
+object HoodieCommonUtils extends Logging {
+
+ def createHoodieClientFromPath(sparkSession: SparkSession, basePath: String,
+ conf: Map[String, String]): SparkRDDWriteClient[_] = {
+ val metaClient = HoodieTableMetaClient.builder().setBasePath(basePath)
+ .setConf(sparkSession.sessionState.newHadoopConf()).build()
+ val schemaUtil = new TableSchemaResolver(metaClient)
+ val schemaStr = schemaUtil.getTableAvroSchemaWithoutMetadataFields.toString
+ val finalParameters = HoodieWriterUtils.parametersWithWriteDefaults(
+ withSparkConf(sparkSession, Map.empty)(
+ conf + (DataSourceWriteOptions.TABLE_TYPE.key() -> metaClient.getTableType.name()))
+ )
+
+ val jsc = new JavaSparkContext(sparkSession.sparkContext)
+ DataSourceUtils.createHoodieClient(jsc, schemaStr, basePath,
+ metaClient.getTableConfig.getTableName, finalParameters.asJava)
+ }
+
+ def getPartitionSchemaFromProperty(metaClient: HoodieTableMetaClient,
+ schemaSpec: Option[StructType]): StructType = {
+ val schema = schemaSpec.getOrElse({
+ val schemaUtil = new TableSchemaResolver(metaClient)
+ AvroConversionUtils.convertAvroSchemaToStructType(schemaUtil.getTableAvroSchema)
+ })
+
+ val tableConfig = metaClient.getTableConfig
+ val partitionColumns = tableConfig.getPartitionFields
+ val nameFieldMap = generateFieldMap(schema)
+
+ if (partitionColumns.isPresent) {
+ // Note that key generator class name could be null
+ val keyGeneratorClassName = tableConfig.getKeyGeneratorClassName
+ if (classOf[TimestampBasedKeyGenerator].getName.equalsIgnoreCase(keyGeneratorClassName)
+ || classOf[TimestampBasedAvroKeyGenerator].getName.equalsIgnoreCase(keyGeneratorClassName)) {
+ val partitionFields = partitionColumns.get().map(column => StructField(column, StringType))
+ StructType(partitionFields)
+ } else {
+ val partitionFields = partitionColumns.get().map(column =>
+ nameFieldMap.getOrElse(column, throw new IllegalArgumentException(s"Cannot find column: '" +
+ s"$column' in the schema[${schema.fields.mkString(",")}]")))
+ StructType(partitionFields)
+ }
+ } else {
+ // If the partition columns have not stored in hoodie.properties(the table that was
+ // created earlier), we trait it as a non-partitioned table.
+ logWarning("No partition columns available from hoodie.properties." +
+ " Partition pruning will not work")
+ new StructType()
+ }
+ }
+
+ /**
+ * This method unravels [[StructType]] into a [[Map]] of pairs of dot-path notation with corresponding
+ * [[StructField]] object for every field of the provided [[StructType]], recursively.
+ *
+ * For example, following struct
+ * <pre>
Review comment:
Can you please make sure that the formatting of the comment is preserved? It become practically unreadable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1058120736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422",
"triggerID" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"triggerType" : "PUSH"
}, {
"hash" : "95158068499ec54bf551d9a03ca315e3fa9fb45d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6512",
"triggerID" : "95158068499ec54bf551d9a03ca315e3fa9fb45d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422)
* 95158068499ec54bf551d9a03ca315e3fa9fb45d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6512)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao merged pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao merged pull request #4901:
URL: https://github.com/apache/hudi/pull/4901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] huberylee commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
huberylee commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054969972
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a change in pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on a change in pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#discussion_r818639313
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/strategy/PartitionAwareClusteringPlanStrategy.java
##########
@@ -113,6 +114,15 @@ public PartitionAwareClusteringPlanStrategy(HoodieTable table, HoodieEngineConte
.build());
}
+ public List<String> getMatchedPartitions(HoodieWriteConfig config, List<String> partitionPaths) {
+ String partitionSelected = config.getClusteringPartitionSelected();
+ if (!StringUtils.isNullOrEmpty(partitionSelected)) {
+ return Arrays.asList(partitionSelected.split(","));
Review comment:
`delimiter`which is comma by default?
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/BaseProcedure.scala
##########
@@ -96,4 +110,40 @@ abstract class BaseProcedure extends Procedure {
throw new UnsupportedOperationException(s"type: ${dataType.typeName} not supported")
}
}
+
+ protected def getBasePath(tableName: Option[Any], tablePath: Option[Any]): String = {
+ tableName.map(
+ t => HoodieCatalogTable(sparkSession, new TableIdentifier(t.asInstanceOf[String])).tableLocation)
+ .getOrElse(
+ tablePath.map(p => p.asInstanceOf[String]).getOrElse(
+ throw new HoodieClusteringException("Table name or table path must be given one"))
+ )
+ }
+
+ protected def convertCatalystType(value: String, dataType: DataType): Any = {
+ try {
+ val valueWithType = dataType match {
+ case StringType => value
+ case BinaryType => value.getBytes(Charset.forName("utf-8"))
+ case BooleanType => value.toBoolean
+ case DoubleType => value.toDouble
+ case _: DecimalType => Decimal(BigDecimal(value))
Review comment:
Does the `precision` of decimal need not be processed?
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowClusteringProcedure.scala
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.util.ClusteringUtils
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types._
+
+import java.util.function.Supplier
+import scala.collection.JavaConverters._
+
+class ShowClusteringProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType, None),
Review comment:
ditto table
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/RunClusteringProcedure.scala
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.BaseHoodieTableFileIndex.PartitionPath
+import org.apache.hudi.client.common.HoodieSparkEngineContext
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig
+import org.apache.hudi.common.table.{HoodieTableMetaClient, TableSchemaResolver}
+import org.apache.hudi.common.util.{ClusteringUtils, Option => HOption}
+import org.apache.hudi.config.HoodieClusteringConfig
+import org.apache.hudi.exception.HoodieClusteringException
+import org.apache.hudi.metadata.HoodieTableMetadata
+import org.apache.hudi.{HoodieCommonUtils, SparkAdapterSupport}
+import org.apache.spark.api.java.JavaSparkContext
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.types._
+
+import java.util.Properties
+import java.util.function.Supplier
+import scala.collection.JavaConverters
+import scala.collection.JavaConverters._
+
+class RunClusteringProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+ /**
+ * OPTIMIZE table_name|table_path [WHERE predicate]
+ * [ORDER BY (col_name1 [, ...] ) ]
+ */
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType, None),
Review comment:
table is also optional，not required？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1050755055
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afde89d0dfa27c145a804d54acfb30b3b21ebc9f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273)
* a25bc26caa76f22f6f0ca91305906b02d50816f6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1051550139
@huberylee Thank you very much for contributing， pls fixed the CI build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1050813470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a25bc26caa76f22f6f0ca91305906b02d50816f6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054972004
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054929568
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fac4277d943e70f8e0375b386bd197b5f4e88c0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399)
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054999877
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410)
* 7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1055117112
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422",
"triggerID" : "7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6422)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] huberylee edited a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
huberylee edited a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054427848
@xiarixiaoyao @XuQianJin-Stars Sorry for the late reply, I will fix it right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054487071
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a25bc26caa76f22f6f0ca91305906b02d50816f6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307)
* fac4277d943e70f8e0375b386bd197b5f4e88c0c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054970151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* fac4277d943e70f8e0375b386bd197b5f4e88c0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399)
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054927965
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fac4277d943e70f8e0375b386bd197b5f4e88c0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399)
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1051549434
@XuQianJin-Stars do you have some time to review this pr, thanks, may be we can add this to HUDI-3161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [WIP][HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1049836222
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afde89d0dfa27c145a804d54acfb30b3b21ebc9f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054547429
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fac4277d943e70f8e0375b386bd197b5f4e88c0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054972004
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Supporting Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1054970151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6273",
"triggerID" : "afde89d0dfa27c145a804d54acfb30b3b21ebc9f",
"triggerType" : "PUSH"
}, {
"hash" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6307",
"triggerID" : "a25bc26caa76f22f6f0ca91305906b02d50816f6",
"triggerType" : "PUSH"
}, {
"hash" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399",
"triggerID" : "fac4277d943e70f8e0375b386bd197b5f4e88c0c",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410",
"triggerID" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"triggerType" : "PUSH"
}, {
"hash" : "ceec40b0d2b0fb64c7288cc79a1bd127e47df822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417",
"triggerID" : "1054969972",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* fac4277d943e70f8e0375b386bd197b5f4e88c0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6399)
* ceec40b0d2b0fb64c7288cc79a1bd127e47df822 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6410) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on pull request #4901:
URL: https://github.com/apache/hudi/pull/4901#issuecomment-1058820753
@huberylee can you please add a description for this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org