You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vickithedeveloper (via GitHub)" <gi...@apache.org> on 2023/04/03 08:43:21 UTC
[GitHub] [hudi] vickithedeveloper opened a new issue, #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError:
vickithedeveloper opened a new issue, #8366:
URL: https://github.com/apache/hudi/issues/8366
I have a simple FLINK streaming code that reads the data from Kafka and write it into a Hudi table. I have followed the exact same code shown in this link - https://hudi.apache.org/docs/flink-quick-start-guide/#quick-start.
It is running fine in my local machine but not running within cluster where there are 2 task managers and 1 job manager.
When submitting the jar File I am getting the following error
`
org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application. at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:113) at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.CompletionException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.configuration.FlinkOptions at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ... 2 more `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions [hudi]
Posted by "ankit0811 (via GitHub)" <gi...@apache.org>.
ankit0811 commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-2045555794
so this is the final pom
```
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner_2.12</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka</artifactId>
<version>${flink.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>3.4.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>${log4j.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>${log4j.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-flink${flink.binary.version}-bundle</artifactId>
<version>${hudi.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-common</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-runtime</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>3.1.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.1.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.1.4</version>
<scope>provided</scope>
</dependency>
</dependencies>
```
and all provided jars are copied to the location `/opt/flink/lib` which is part of the class path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1498452922
What kind of options do you wanna config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vickithedeveloper commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions
Posted by "vickithedeveloper (via GitHub)" <gi...@apache.org>.
vickithedeveloper commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1501596092
@danny0405 option to configure the access key for the azure storage accounts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError:
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1495439988
Which class is missing here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1569713838
@vickithedeveloper Were you able to fix these library conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vickithedeveloper commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions
Posted by "vickithedeveloper (via GitHub)" <gi...@apache.org>.
vickithedeveloper commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1497771034
@danny0405 I have provided base path as "wasbs: //container@storageAccountNmae.blob.core.windows.net"
When submitting the job to FLINK I am getting a no credential exception. How can I pass the value to Hudi?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vickithedeveloper commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError:
Posted by "vickithedeveloper (via GitHub)" <gi...@apache.org>.
vickithedeveloper commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1494053461
@danny0405 could you please provide more insight into this and how I can solve it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions [hudi]
Posted by "ankit0811 (via GitHub)" <gi...@apache.org>.
ankit0811 commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-2038338671
Thanks for responding @danny0405 This is what the class path logs on the taskManager
```
Classpath: /opt/flink/lib/flink-cep-1.15.2.jar:/opt/flink/lib/flink-connector-files-1.15.2.jar:/opt/flink/lib/flink-csv-1.15.2.jar:/opt/flink/lib/flink-gs-fs-hadoop-1.15.2.jar:/opt/flink/lib/flink-json-1.15.2.jar:/opt/flink/lib/flink-queryable-state-runtime-1.15.2.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.15.2.jar:/opt/flink/lib/flink-s3-fs-presto-1.15.2.jar:/opt/flink/lib/flink-scala_2.12-1.15.2.jar:/opt/flink/lib/flink-shaded-zookeeper-3.5.9.jar:/opt/flink/lib/flink-table-api-java-uber-1.15.2.jar:/opt/flink/lib/flink-table-planner-loader-1.15.2.jar:/opt/flink/lib/flink-table-runtime-1.15.2.jar:/opt/flink/lib/hadoop-client-3.2.1.jar:/opt/flink/lib/hudi-flink1.15-bundle-0.14.0.jar:/opt/flink/lib/jackson-annotations-2.9.0.jar:/opt/flink/lib/jackson-core-2.9.8.jar:/opt/flink/lib/jackson-databind-2.9.8.jar:/opt/flink/lib/jna-5.4.0.jar:/opt/flink/lib/jna-platform-5.4.0.jar:/opt/flink/lib/log4j-1.2-api-2.17.1.jar:/opt/flink/lib/log4j-api-2.17.1.jar:/opt/flink/lib/log4j-core-2.17.1.jar:/o
pt/flink/lib/log4j-slf4j-impl-2.17.1.jar:/opt/flink/lib/oshi-core-3.4.0.jar:/opt/flink/lib/flink-dist-1.15.2.jar
```
I do see the hudi-flink-bundle there. Any idea what might be wrong
FYI, now I am getting the following exception
```
java.util.concurrent.CompletionException: java.lang.NoSuchFieldError: DATABASE_NAME\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)\n\t... 2 more\nCaused by: java.lang.NoSuchFieldError: DATABASE_NAME\n\tat org.apache.hudi.configuration.FlinkOptions.<clinit>(FlinkOptions.java:88)
```
which I still believe is related. Any help will be much appreciated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-2038561594
```java
java.util.concurrent.CompletionException: java.lang.NoSuchFieldError: DATABASE_NAME\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)\n\t... 2 more\nCaused by: java.lang.NoSuchFieldError: DATABASE_NAME\n\tat org.apache.hudi.configuration.FlinkOptions.<clinit>(FlinkOptions.java:88)
```
This should be a JDK incompatibility issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vickithedeveloper commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions
Posted by "vickithedeveloper (via GitHub)" <gi...@apache.org>.
vickithedeveloper commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1497436403
The above error has been resolved by changing the FLINK version to 1.15.1 and adding hudi-hadoop-mr-bundle dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1496953281
This class should be included in your jar right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vickithedeveloper commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions
Posted by "vickithedeveloper (via GitHub)" <gi...@apache.org>.
vickithedeveloper commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1497433723
yes, I have copied the jar to /opt/flink/lib/ inside the container and jar is loaded by Flink also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vickithedeveloper commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError:
Posted by "vickithedeveloper (via GitHub)" <gi...@apache.org>.
vickithedeveloper commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1495711599
@danny0405 getting error for the following class
`java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.configuration.FlinkOptions`
Using FLINK 1.15.0 and Hudi 0.13.0
In my local the job is running without any issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-2035876822
It looks like the hudi flink bundle jar is not correctly loaded in the classpath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions [hudi]
Posted by "ankit0811 (via GitHub)" <gi...@apache.org>.
ankit0811 commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-2035796844
piggybacking on this issue
@danny0405 I still see this exception thrown when running the listed example for flink version `1.15.2` and hudi version `0.14.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions [hudi]
Posted by "ankit0811 (via GitHub)" <gi...@apache.org>.
ankit0811 commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-2044207518
This occurs when I don't pass the `hudi-flink-bundle` in the class path explicitly. So I am not sure about the jdk incompatibility.
To summarize, if I pass the bundle explicitly in the class path `(/opt/flink/lib/)` and update the pom as
```
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-flink${flink.binary.version}-bundle</artifactId>
<version>${hudi.version}</version>
<scope>provided</scope>
</dependency>
```
, it throws
```
Caused by: java.util.concurrent.CompletionException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.configuration.FlinkOptions at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ... 2 more
```
And if I change the pom dependency for the bundle to be default and not provided
```
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-flink${flink.binary.version}-bundle</artifactId>
<version>${hudi.version}</version>
</dependency>
```
I get the following exception
```
java.util.concurrent.CompletionException: java.lang.NoSuchFieldError: DATABASE_NAME\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)\n\t... 2 more\nCaused by: java.lang.NoSuchFieldError: DATABASE_NAME\n\tat org.apache.hudi.configuration.FlinkOptions.<clinit>(FlinkOptions.java:88)
```
Any pointers will be much appreciated @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-2044698367
> To summarize, if I pass the bundle explicitly in the class path (/opt/flink/lib/) and update the pom as
For most of the cases, the scope should be declared as "provided", and where is the flink bundle jar finally located?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions [hudi]
Posted by "zhangxiongbiao (via GitHub)" <gi...@apache.org>.
zhangxiongbiao commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-2264791197
@ankit0811 I encountered a similar issue as well, and upon thorough debugging, I discovered that there was a conflict between the `hudi-flink${flink.binary.version}-bundle` and the `flink-s3-fs-presto` libraries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1502598669
It's like an avro clazz conflict, mulple avro jars are in the classpath, for Flink, we suggest to use the avro 0.10.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vickithedeveloper commented on issue #8366: [SUPPORT] Flink streaming write to Hudi table using data stream API java.lang.NoClassDefFoundError: org.apache.hudi.configuration.FlinkOptions
Posted by "vickithedeveloper (via GitHub)" <gi...@apache.org>.
vickithedeveloper commented on issue #8366:
URL: https://github.com/apache/hudi/issues/8366#issuecomment-1501596530
@danny0405 getting the following class Not found error
Caused by: java.lang.NoSuchMethodError: org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeAvroMetadata([BLjava/lang/Class;)Lorg/apache/hudi/org/apache/avro/specific/SpecificRecordBase;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org