You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/07 23:38:15 UTC
[GitHub] [hudi] CTTY opened a new pull request, #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
CTTY opened a new pull request, #6628:
URL: https://github.com/apache/hudi/pull/6628
### Change Logs
Make Avro version consistent across Hudi and make sure flink bundle is usable even when user is building against spark3 profile
### Impact
_Describe any public API or user-facing feature change or any performance impact._
**Risk level: none | low | medium | high**
_Choose one. If medium or high, explain what verification was done to mitigate the risks._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6628:
URL: https://github.com/apache/hudi/pull/6628#issuecomment-1240039267
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2504fd6b17a7a3fb2a77f755d7fe6b6c7f83c96f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11232",
"triggerID" : "2504fd6b17a7a3fb2a77f755d7fe6b6c7f83c96f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2504fd6b17a7a3fb2a77f755d7fe6b6c7f83c96f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11232)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CTTY commented on a diff in pull request #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
Posted by GitBox <gi...@apache.org>.
CTTY commented on code in PR #6628:
URL: https://github.com/apache/hudi/pull/6628#discussion_r968961398
##########
packaging/hudi-flink-bundle/pom.xml:
##########
@@ -501,8 +501,7 @@
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
- <!-- Override the version to be same with Flink avro -->
- <version>1.10.0</version>
+ <version>${avro.version}</version>
Review Comment:
Yes, we've tested with Flink streamer loading data from Kafka datasource in Hudi format. And it works fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CTTY commented on a diff in pull request #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
Posted by GitBox <gi...@apache.org>.
CTTY commented on code in PR #6628:
URL: https://github.com/apache/hudi/pull/6628#discussion_r977943268
##########
packaging/hudi-flink-bundle/pom.xml:
##########
@@ -501,8 +501,7 @@
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
- <!-- Override the version to be same with Flink avro -->
- <version>1.10.0</version>
+ <version>${avro.version}</version>
Review Comment:
Yes, you can revert this avro version change to unblock yourself for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6628:
URL: https://github.com/apache/hudi/pull/6628#discussion_r978172464
##########
packaging/hudi-flink-bundle/pom.xml:
##########
@@ -501,8 +501,7 @@
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
- <!-- Override the version to be same with Flink avro -->
- <version>1.10.0</version>
+ <version>${avro.version}</version>
Review Comment:
Can we fire a fix here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6628:
URL: https://github.com/apache/hudi/pull/6628#discussion_r965422148
##########
packaging/hudi-flink-bundle/pom.xml:
##########
@@ -501,8 +501,7 @@
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
- <!-- Override the version to be same with Flink avro -->
- <version>1.10.0</version>
+ <version>${avro.version}</version>
Review Comment:
Make sure that there are no compatibility issues in flink side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6628:
URL: https://github.com/apache/hudi/pull/6628#issuecomment-1240215510
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2504fd6b17a7a3fb2a77f755d7fe6b6c7f83c96f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11232",
"triggerID" : "2504fd6b17a7a3fb2a77f755d7fe6b6c7f83c96f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2504fd6b17a7a3fb2a77f755d7fe6b6c7f83c96f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11232)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hackergin commented on a diff in pull request #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
Posted by GitBox <gi...@apache.org>.
hackergin commented on code in PR #6628:
URL: https://github.com/apache/hudi/pull/6628#discussion_r971586922
##########
packaging/hudi-flink-bundle/pom.xml:
##########
@@ -501,8 +501,7 @@
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
- <!-- Override the version to be same with Flink avro -->
- <version>1.10.0</version>
+ <version>${avro.version}</version>
Review Comment:
> Yes, we've tested with Flink streamer loading data from Kafka datasource in Hudi format. And it works fine
Hi, @CTTY I met a java.lang.ClassNotFoundException when using the latest master code. Class org.apache.avro.LogicalTypes$LocalTimestampMillis seems to only appear in avro 1.10 version. Please help to confirm this problem, correct me if I am wrong .
```
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.org.apache.avro.LogicalTypes$LocalTimestampMillis
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_202]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_202]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_202]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_202]
at org.apache.hudi.table.HoodieTableFactory.inferAvroSchema(HoodieTableFactory.java:346) ~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
Posted by GitBox <gi...@apache.org>.
yihua merged PR #6628:
URL: https://github.com/apache/hudi/pull/6628
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hackergin commented on a diff in pull request #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
Posted by GitBox <gi...@apache.org>.
hackergin commented on code in PR #6628:
URL: https://github.com/apache/hudi/pull/6628#discussion_r971586922
##########
packaging/hudi-flink-bundle/pom.xml:
##########
@@ -501,8 +501,7 @@
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
- <!-- Override the version to be same with Flink avro -->
- <version>1.10.0</version>
+ <version>${avro.version}</version>
Review Comment:
> Yes, we've tested with Flink streamer loading data from Kafka datasource in Hudi format. And it works fine
Hi, @CTTY I met a java.lang.ClassNotFoundException when using the latest master code. Class org.apache.avro.LogicalTypes$LocalTimestampMillis seems to only appear in avro 1.10 version. Please help to confirm this problem, correct me if I am wrong .
```
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.org.apache.avro.LogicalTypes$LocalTimestampMillis
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_202]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_202]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_202]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_202]
at org.apache.hudi.table.HoodieTableFactory.inferAvroSchema(HoodieTableFactory.java:346) ~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6628:
URL: https://github.com/apache/hudi/pull/6628#discussion_r968965667
##########
packaging/hudi-flink-bundle/pom.xml:
##########
@@ -501,8 +501,7 @@
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
- <!-- Override the version to be same with Flink avro -->
- <version>1.10.0</version>
+ <version>${avro.version}</version>
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6628: [HUDI-4806] Use Avro version from the root pom for Flink bundle
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6628:
URL: https://github.com/apache/hudi/pull/6628#issuecomment-1240035433
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2504fd6b17a7a3fb2a77f755d7fe6b6c7f83c96f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2504fd6b17a7a3fb2a77f755d7fe6b6c7f83c96f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2504fd6b17a7a3fb2a77f755d7fe6b6c7f83c96f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org