You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/22 16:48:14 UTC
[GitHub] [hudi] codope opened a new pull request, #6472: [HUDI-4549] Remove Avro shading from hudi-hive-sync-bundle
codope opened a new pull request, #6472:
URL: https://github.com/apache/hudi/pull/6472
### Change Logs
- Remove Avro shading from hudi-hive-sync-bundle
- We cannot shade avro in hudi-spark-bundle or hudi-utilities-bundle because we use certain SparkConf APIs that expect unshaded Avro classes. If we shade Avro in hudi-hive-sync-bundle without shading in hudi-spark or hudi-utilities bundle then it can lead to conflict (NoMethodFoundError) as mentioned in HUDI-4549.
### Impact
_Describe any public API or user-facing feature change or any performance impact._
**Risk level: none | low | medium | high**
Tested hive sync locally and on EMR with changing the order of jars in the launch command.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6472: [HUDI-4549] Remove Avro shading from hudi-hive-sync-bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6472:
URL: https://github.com/apache/hudi/pull/6472#issuecomment-1222704115
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "faecb216bdeb30a459040846bb9a5167556fd605",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879",
"triggerID" : "faecb216bdeb30a459040846bb9a5167556fd605",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* faecb216bdeb30a459040846bb9a5167556fd605 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6472: [HUDI-4549] Remove avro from hudi-hive-sync-bundle and hudi-aws-bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6472:
URL: https://github.com/apache/hudi/pull/6472#issuecomment-1229135604
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "faecb216bdeb30a459040846bb9a5167556fd605",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879",
"triggerID" : "faecb216bdeb30a459040846bb9a5167556fd605",
"triggerType" : "PUSH"
}, {
"hash" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10984",
"triggerID" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"triggerType" : "PUSH"
}, {
"hash" : "1f66caeefe75634af69cddb9bae37bd0e469c7cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10989",
"triggerID" : "1f66caeefe75634af69cddb9bae37bd0e469c7cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02329634ac100d362b6e9fa714faaad3e27298f4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10984)
* 1f66caeefe75634af69cddb9bae37bd0e469c7cb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10989)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6472: [HUDI-4549] Remove avro from hudi-hive-sync-bundle and hudi-aws-bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6472:
URL: https://github.com/apache/hudi/pull/6472#issuecomment-1229147982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "faecb216bdeb30a459040846bb9a5167556fd605",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879",
"triggerID" : "faecb216bdeb30a459040846bb9a5167556fd605",
"triggerType" : "PUSH"
}, {
"hash" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10984",
"triggerID" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"triggerType" : "PUSH"
}, {
"hash" : "1f66caeefe75634af69cddb9bae37bd0e469c7cb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10989",
"triggerID" : "1f66caeefe75634af69cddb9bae37bd0e469c7cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1f66caeefe75634af69cddb9bae37bd0e469c7cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10989)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #6472: [HUDI-4549] Remove avro from hudi-hive-sync-bundle and hudi-aws-bundle
Posted by GitBox <gi...@apache.org>.
codope merged PR #6472:
URL: https://github.com/apache/hudi/pull/6472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6472: [HUDI-4549] Remove Avro shading from hudi-hive-sync-bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6472:
URL: https://github.com/apache/hudi/pull/6472#issuecomment-1228883212
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "faecb216bdeb30a459040846bb9a5167556fd605",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879",
"triggerID" : "faecb216bdeb30a459040846bb9a5167556fd605",
"triggerType" : "PUSH"
}, {
"hash" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* faecb216bdeb30a459040846bb9a5167556fd605 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879)
* 02329634ac100d362b6e9fa714faaad3e27298f4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6472: [HUDI-4549] Remove avro from hudi-hive-sync-bundle and hudi-aws-bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6472:
URL: https://github.com/apache/hudi/pull/6472#issuecomment-1229135105
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "faecb216bdeb30a459040846bb9a5167556fd605",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879",
"triggerID" : "faecb216bdeb30a459040846bb9a5167556fd605",
"triggerType" : "PUSH"
}, {
"hash" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10984",
"triggerID" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"triggerType" : "PUSH"
}, {
"hash" : "1f66caeefe75634af69cddb9bae37bd0e469c7cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1f66caeefe75634af69cddb9bae37bd0e469c7cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02329634ac100d362b6e9fa714faaad3e27298f4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10984)
* 1f66caeefe75634af69cddb9bae37bd0e469c7cb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6472: [HUDI-4549] Remove Avro shading from hudi-hive-sync-bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6472:
URL: https://github.com/apache/hudi/pull/6472#issuecomment-1229002706
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "faecb216bdeb30a459040846bb9a5167556fd605",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879",
"triggerID" : "faecb216bdeb30a459040846bb9a5167556fd605",
"triggerType" : "PUSH"
}, {
"hash" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10984",
"triggerID" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02329634ac100d362b6e9fa714faaad3e27298f4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10984)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6472: [HUDI-4549] Remove avro from hudi-hive-sync-bundle and hudi-aws-bundle
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6472:
URL: https://github.com/apache/hudi/pull/6472#discussion_r975834051
##########
packaging/hudi-aws-bundle/pom.xml:
##########
@@ -284,19 +275,13 @@
<artifactId>hudi-aws</artifactId>
<version>${project.version}</version>
</dependency>
- <!-- Need parquet and avro to run AwsGlueCatalogSyncTool using run_sync_tool with this bundle.
- Parquet and avro from other packages have already been shaded above-->
+
+ <!-- Parquet -->
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-avro</artifactId>
<version>${parquet.version}</version>
<scope>compile</scope>
</dependency>
- <dependency>
- <groupId>org.apache.avro</groupId>
Review Comment:
Shouldn't we do the same for GCP bundle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6472: [HUDI-4549] Remove Avro shading from hudi-hive-sync-bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6472:
URL: https://github.com/apache/hudi/pull/6472#issuecomment-1222881613
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "faecb216bdeb30a459040846bb9a5167556fd605",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879",
"triggerID" : "faecb216bdeb30a459040846bb9a5167556fd605",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* faecb216bdeb30a459040846bb9a5167556fd605 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #6472: [HUDI-4549] Remove avro from hudi-hive-sync-bundle and hudi-aws-bundle
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #6472:
URL: https://github.com/apache/hudi/pull/6472#issuecomment-1229131453
@codope i tested running hive sync tool and glue sync tool alone from both bundles in emr env. worked fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6472: [HUDI-4549] Remove Avro shading from hudi-hive-sync-bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6472:
URL: https://github.com/apache/hudi/pull/6472#issuecomment-1228886953
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "faecb216bdeb30a459040846bb9a5167556fd605",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879",
"triggerID" : "faecb216bdeb30a459040846bb9a5167556fd605",
"triggerType" : "PUSH"
}, {
"hash" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10984",
"triggerID" : "02329634ac100d362b6e9fa714faaad3e27298f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* faecb216bdeb30a459040846bb9a5167556fd605 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10879)
* 02329634ac100d362b6e9fa714faaad3e27298f4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10984)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #6472: [HUDI-4549] Remove avro from hudi-hive-sync-bundle and hudi-aws-bundle
Posted by GitBox <gi...@apache.org>.
codope commented on PR #6472:
URL: https://github.com/apache/hudi/pull/6472#issuecomment-1229435732
> @codope i tested running hive sync tool and glue sync tool alone from both bundles in emr env. worked fine.
Sounds good. I'll land it then. The failing test is flaky and there is already a ticket to fix it HUDI-4327.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6472: [HUDI-4549] Remove Avro shading from hudi-hive-sync-bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6472:
URL: https://github.com/apache/hudi/pull/6472#issuecomment-1222691832
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "faecb216bdeb30a459040846bb9a5167556fd605",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "faecb216bdeb30a459040846bb9a5167556fd605",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* faecb216bdeb30a459040846bb9a5167556fd605 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #6472: [HUDI-4549] Remove avro from hudi-hive-sync-bundle and hudi-aws-bundle
Posted by GitBox <gi...@apache.org>.
codope commented on code in PR #6472:
URL: https://github.com/apache/hudi/pull/6472#discussion_r976268767
##########
packaging/hudi-aws-bundle/pom.xml:
##########
@@ -284,19 +275,13 @@
<artifactId>hudi-aws</artifactId>
<version>${project.version}</version>
</dependency>
- <!-- Need parquet and avro to run AwsGlueCatalogSyncTool using run_sync_tool with this bundle.
- Parquet and avro from other packages have already been shaded above-->
+
+ <!-- Parquet -->
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-avro</artifactId>
<version>${parquet.version}</version>
<scope>compile</scope>
</dependency>
- <dependency>
- <groupId>org.apache.avro</groupId>
Review Comment:
You're right. The same reasoning should apply to gcp bundle as well. HUDI-4889 to track.
I am targeting it for 0.13.0 for the lack of time to test the changes right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org