You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/23 15:55:06 UTC
[GitHub] [hudi] codope opened a new pull request, #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
codope opened a new pull request, #5666:
URL: https://github.com/apache/hudi/pull/5666
## What is the purpose of the pull request
This PR attempts to decouple `hudi-aws` from `hudi-client-common`. It is being used in two places:
1. During building of hoodie write config (metrics config) for commit executors or metadata writer.
2. During meta-sync in hudi-flink if sync mode uses Glue.
## Brief change log
* Make hudi-aws a runtime dependency in hudi-client-common and hudi-flink.
* Add builder for AwsGlueCatalogSyncTool.
## Verify this pull request
*(Please pick either of the following options)*
This PR is mainly refactoring and dependency changes and it should be covered with existing tests.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5666:
URL: https://github.com/apache/hudi/pull/5666#issuecomment-1134856648
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "145542b26e6dae731fb155ee048073c568065941",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "145542b26e6dae731fb155ee048073c568065941",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 145542b26e6dae731fb155ee048073c568065941 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #5666:
URL: https://github.com/apache/hudi/pull/5666#discussion_r880435101
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AwsGlueCatalogSyncTool.java:
##########
@@ -65,6 +63,56 @@ public static void main(String[] args) {
FileSystem fs = FSUtils.getFs(cfg.basePath, new Configuration());
HiveConf hiveConf = new HiveConf();
hiveConf.addResource(fs.getConf());
- new AwsGlueCatalogSyncTool(cfg, hiveConf, fs).syncHoodieTable();
+ AwsGlueCatalogSyncTool.newBuilder()
Review Comment:
sounds good. we may also incorporate the builder pattern in the metasync refactoring work with @fengjian428
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5666:
URL: https://github.com/apache/hudi/pull/5666#issuecomment-1136170317
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "145542b26e6dae731fb155ee048073c568065941",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8869",
"triggerID" : "145542b26e6dae731fb155ee048073c568065941",
"triggerType" : "PUSH"
}, {
"hash" : "d679d17404397e420fea70dbb5449e299ab66a76",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8897",
"triggerID" : "d679d17404397e420fea70dbb5449e299ab66a76",
"triggerType" : "PUSH"
}, {
"hash" : "f9a7a610284b8cefe52fcec55e52f0ea10a01b3e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8902",
"triggerID" : "f9a7a610284b8cefe52fcec55e52f0ea10a01b3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9a7a610284b8cefe52fcec55e52f0ea10a01b3e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8902)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5666:
URL: https://github.com/apache/hudi/pull/5666#issuecomment-1134860849
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "145542b26e6dae731fb155ee048073c568065941",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8869",
"triggerID" : "145542b26e6dae731fb155ee048073c568065941",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 145542b26e6dae731fb155ee048073c568065941 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8869)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
codope commented on code in PR #5666:
URL: https://github.com/apache/hudi/pull/5666#discussion_r880314331
##########
hudi-flink-datasource/hudi-flink/pom.xml:
##########
@@ -89,6 +89,12 @@
<artifactId>hudi-client-common</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-aws</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
Review Comment:
The `provided` dependencies in hudi-client-common should not be included in `hudi-flink`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5666:
URL: https://github.com/apache/hudi/pull/5666#issuecomment-1135031691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "145542b26e6dae731fb155ee048073c568065941",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8869",
"triggerID" : "145542b26e6dae731fb155ee048073c568065941",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 145542b26e6dae731fb155ee048073c568065941 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8869)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #5666:
URL: https://github.com/apache/hudi/pull/5666#discussion_r880378853
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AwsGlueCatalogSyncTool.java:
##########
@@ -65,6 +63,56 @@ public static void main(String[] args) {
FileSystem fs = FSUtils.getFs(cfg.basePath, new Configuration());
HiveConf hiveConf = new HiveConf();
hiveConf.addResource(fs.getConf());
- new AwsGlueCatalogSyncTool(cfg, hiveConf, fs).syncHoodieTable();
+ AwsGlueCatalogSyncTool.newBuilder()
Review Comment:
the builder pattern is good. but do you need it be in this PR? i was only expecting pom.xml changes :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
codope merged PR #5666:
URL: https://github.com/apache/hudi/pull/5666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5666:
URL: https://github.com/apache/hudi/pull/5666#issuecomment-1135966032
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "145542b26e6dae731fb155ee048073c568065941",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8869",
"triggerID" : "145542b26e6dae731fb155ee048073c568065941",
"triggerType" : "PUSH"
}, {
"hash" : "d679d17404397e420fea70dbb5449e299ab66a76",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8897",
"triggerID" : "d679d17404397e420fea70dbb5449e299ab66a76",
"triggerType" : "PUSH"
}, {
"hash" : "f9a7a610284b8cefe52fcec55e52f0ea10a01b3e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8902",
"triggerID" : "f9a7a610284b8cefe52fcec55e52f0ea10a01b3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d679d17404397e420fea70dbb5449e299ab66a76 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8897)
* f9a7a610284b8cefe52fcec55e52f0ea10a01b3e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8902)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5666:
URL: https://github.com/apache/hudi/pull/5666#discussion_r880015739
##########
hudi-flink-datasource/hudi-flink/pom.xml:
##########
@@ -89,6 +89,12 @@
<artifactId>hudi-client-common</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-aws</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
Review Comment:
We still add a dependency for the `hudi-client-common`, if we want to avoid clazz conflicts, a better solution is to create a shaded module for these vendor jars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5666:
URL: https://github.com/apache/hudi/pull/5666#issuecomment-1135817505
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "145542b26e6dae731fb155ee048073c568065941",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8869",
"triggerID" : "145542b26e6dae731fb155ee048073c568065941",
"triggerType" : "PUSH"
}, {
"hash" : "d679d17404397e420fea70dbb5449e299ab66a76",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d679d17404397e420fea70dbb5449e299ab66a76",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 145542b26e6dae731fb155ee048073c568065941 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8869)
* d679d17404397e420fea70dbb5449e299ab66a76 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
codope commented on code in PR #5666:
URL: https://github.com/apache/hudi/pull/5666#discussion_r880313503
##########
hudi-flink-datasource/hudi-flink/pom.xml:
##########
@@ -89,6 +89,12 @@
<artifactId>hudi-client-common</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-aws</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
Review Comment:
I was going to take it up after this patch lands. HUDI-4011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5666:
URL: https://github.com/apache/hudi/pull/5666#issuecomment-1135820940
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "145542b26e6dae731fb155ee048073c568065941",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8869",
"triggerID" : "145542b26e6dae731fb155ee048073c568065941",
"triggerType" : "PUSH"
}, {
"hash" : "d679d17404397e420fea70dbb5449e299ab66a76",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8897",
"triggerID" : "d679d17404397e420fea70dbb5449e299ab66a76",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 145542b26e6dae731fb155ee048073c568065941 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8869)
* d679d17404397e420fea70dbb5449e299ab66a76 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8897)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
codope commented on code in PR #5666:
URL: https://github.com/apache/hudi/pull/5666#discussion_r880432062
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AwsGlueCatalogSyncTool.java:
##########
@@ -65,6 +63,56 @@ public static void main(String[] args) {
FileSystem fs = FSUtils.getFs(cfg.basePath, new Configuration());
HiveConf hiveConf = new HiveConf();
hiveConf.addResource(fs.getConf());
- new AwsGlueCatalogSyncTool(cfg, hiveConf, fs).syncHoodieTable();
+ AwsGlueCatalogSyncTool.newBuilder()
Review Comment:
I can move it to a separate PR. I just noticed and did the change. But, better to keep it in a separate PR just in case if we have to revert this PR then we only revert dependency changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5666: [HUDI-3193] Decouple hudi-aws from hudi-client-common
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5666:
URL: https://github.com/apache/hudi/pull/5666#issuecomment-1135961449
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "145542b26e6dae731fb155ee048073c568065941",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8869",
"triggerID" : "145542b26e6dae731fb155ee048073c568065941",
"triggerType" : "PUSH"
}, {
"hash" : "d679d17404397e420fea70dbb5449e299ab66a76",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8897",
"triggerID" : "d679d17404397e420fea70dbb5449e299ab66a76",
"triggerType" : "PUSH"
}, {
"hash" : "f9a7a610284b8cefe52fcec55e52f0ea10a01b3e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f9a7a610284b8cefe52fcec55e52f0ea10a01b3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d679d17404397e420fea70dbb5449e299ab66a76 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8897)
* f9a7a610284b8cefe52fcec55e52f0ea10a01b3e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org