You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/08/16 14:39:45 UTC
[GitHub] [hudi] izhangzhihao opened a new issue #3484: [SUPPORT] ClassNotFoundException: org.apache.hudi.sink.StreamWriteOperatorCoordinator$Provider
izhangzhihao opened a new issue #3484:
URL: https://github.com/apache/hudi/issues/3484
Steps to reproduce the behavior:
1. build hudi flink stuff from master https://github.com/apache/hudi/commit/66f951322a3872073b86896fa5c10b51a0f6e4ab
2.
```
CREATE TABLE t1
(
uuid VARCHAR(20), -- you can use 'PRIMARY KEY NOT ENFORCED' syntax to mark the field as record key
name VARCHAR(10),
age INT,
ts TIMESTAMP(3),
`partition` VARCHAR(20)
) PARTITIONED BY (`partition`) WITH (
'connector' = 'hudi',
'path' = '/data/t1',
'write.tasks' = '1', -- default is 4 ,required more resource
'compaction.tasks' = '1', -- default is 10 ,required more resource
'table.type' = 'COPY_ON_WRITE', -- this creates a MERGE_ON_READ table, by default is COPY_ON_WRITE
'read.tasks' = '1', -- default is 4 ,required more resource
'read.streaming.enabled' = 'true', -- this option enable the streaming read
'read.streaming.start-commit' = '20210712134429', -- specifies the start commit instant time
'read.streaming.check-interval' = '4' -- specifies the check interval for finding new source commits, default 60s.
);
```
3.
```
INSERT INTO t1
VALUES ('id1', 'Danny', 23, TIMESTAMP '1970-01-01 00:00:01', 'par1'),
('id2', 'Stephen', 33, TIMESTAMP '1970-01-01 00:00:02', 'par1'),
('id3', 'Julian', 53, TIMESTAMP '1970-01-01 00:00:03', 'par2'),
('id4', 'Fabian', 31, TIMESTAMP '1970-01-01 00:00:04', 'par2'),
('id5', 'Sophia', 18, TIMESTAMP '1970-01-01 00:00:05', 'par3'),
('id6', 'Emma', 20, TIMESTAMP '1970-01-01 00:00:06', 'par3'),
('id7', 'Bob', 44, TIMESTAMP '1970-01-01 00:00:07', 'par4'),
('id8', 'Han', 56, TIMESTAMP '1970-01-01 00:00:08', 'par4');
```
4. error happened...
5. The file `hudi-flink-bundle_2.12-0.10.0-SNAPSHOT.jar` & `hudi-utilities-bundle_2.12-0.10.0-SNAPSHOT.jar` do exist in `/opt/flink/lib/` of the JobManager
**Expected behavior**
without any error
**Environment Description**
* Hudi version : master
* Flink version : 1.13.1
* Hive version : none
* Hadoop version : 2.9.2
* Storage (HDFS/S3/GCS..) : local file
* Running on Docker? (yes/no) : yes
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
Job failed during initialization of JobManager
org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster.
at org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: org.apache.flink.runtime.JobException: Cannot instantiate the coordinator for operator hoodie_stream_write
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
... 7 more
Caused by: java.lang.RuntimeException: org.apache.flink.runtime.JobException: Cannot instantiate the coordinator for operator hoodie_stream_write
at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:316)
at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
... 7 more
Caused by: org.apache.flink.runtime.JobException: Cannot instantiate the coordinator for operator hoodie_stream_write
at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:217)
at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.attachJobGraph(DefaultExecutionGraph.java:792)
at org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph(DefaultExecutionGraphBuilder.java:196)
at org.apache.flink.runtime.scheduler.DefaultExecutionGraphFactory.createAndRestoreExecutionGraph(DefaultExecutionGraphFactory.java:107)
at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:342)
at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:190)
at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:120)
at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:132)
at org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:110)
at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:340)
at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:317)
at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:107)
at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:95)
at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
... 7 more
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.sink.StreamWriteOperatorCoordinator$Provider
at java.base/java.net.URLClassLoader.findClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:172)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Unknown Source)
at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:76)
at java.base/java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
at java.base/java.io.ObjectInputStream.readClassDesc(Unknown Source)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
at java.base/java.io.ObjectInputStream.readObject(Unknown Source)
at java.base/java.io.ObjectInputStream.readObject(Unknown Source)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587)
at org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.create(OperatorCoordinatorHolder.java:435)
at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:211)
... 20 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #3484: [SUPPORT] ClassNotFoundException: org.apache.hudi.sink.StreamWriteOperatorCoordinator$Provider
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #3484:
URL: https://github.com/apache/hudi/issues/3484#issuecomment-899963301
Weird, can you show other libraries together with the hudi bundle jar ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] izhangzhihao closed issue #3484: [SUPPORT] ClassNotFoundException: org.apache.hudi.sink.StreamWriteOperatorCoordinator$Provider
Posted by GitBox <gi...@apache.org>.
izhangzhihao closed issue #3484:
URL: https://github.com/apache/hudi/issues/3484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] izhangzhihao commented on issue #3484: [SUPPORT] ClassNotFoundException: org.apache.hudi.sink.StreamWriteOperatorCoordinator$Provider
Posted by GitBox <gi...@apache.org>.
izhangzhihao commented on issue #3484:
URL: https://github.com/apache/hudi/issues/3484#issuecomment-899965860
Ah, it seems missing permission...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] izhangzhihao commented on issue #3484: [SUPPORT] ClassNotFoundException: org.apache.hudi.sink.StreamWriteOperatorCoordinator$Provider
Posted by GitBox <gi...@apache.org>.
izhangzhihao commented on issue #3484:
URL: https://github.com/apache/hudi/issues/3484#issuecomment-899965181
```
root@08f97b702585:/opt/flink# ls /opt/flink/lib -al
total 396328
drwxrwxr-x 1 flink flink 4096 Aug 16 14:29 .
drwxr-xr-x 1 flink flink 4096 Jun 2 03:02 ..
-rw-r--r-- 1 root root 6148 Aug 16 06:18 .DS_Store
-rw-r--r-- 1 root root 61829 Jul 23 2018 commons-logging-1.2.jar
-rw-r--r-- 1 root root 26777311 Aug 16 07:51 flink-connector-postgres-cdc-2.1-SNAPSHOT.jar
-rw-r--r-- 1 flink flink 92311 May 25 12:13 flink-csv-1.13.1.jar
-rw-r--r-- 1 flink flink 106648120 May 25 12:16 flink-dist_2.12-1.13.1.jar
-rw-r--r-- 1 flink flink 148131 May 25 12:12 flink-json-1.13.1.jar
-rw-r--r-- 1 root root 43317025 Feb 24 2020 flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
-rw-rw-r-- 1 flink flink 7709740 Apr 8 14:08 flink-shaded-zookeeper-3.4.14.jar
-rw-r--r-- 1 root root 27764261 Jun 7 09:51 flink-sql-connector-elasticsearch7_2.12-1.13.1.jar
-rw-r--r-- 1 flink flink 38533349 May 25 12:16 flink-table-blink_2.12-1.13.1.jar
-rw-r--r-- 1 flink flink 35015429 May 25 12:16 flink-table_2.12-1.13.1.jar
-rw-r--r-- 1 root root 25521267 Jul 4 09:24 hadoop-common-shaded-SNAPSHOT.jar
-rw------- 1 root root 50991633 Aug 16 14:24 hudi-flink-bundle_2.12-0.10.0-SNAPSHOT.jar
-rw------- 1 root root 40224980 Aug 16 14:29 hudi-utilities-bundle_2.12-0.10.0-SNAPSHOT.jar
-rw-rw-r-- 1 flink flink 67114 Oct 10 2019 log4j-1.2-api-2.12.1.jar
-rw-rw-r-- 1 flink flink 276771 Oct 10 2019 log4j-api-2.12.1.jar
-rw-rw-r-- 1 flink flink 1674433 Oct 10 2019 log4j-core-2.12.1.jar
-rw-rw-r-- 1 flink flink 23518 Oct 10 2019 log4j-slf4j-impl-2.12.1.jar
-rw-r--r-- 1 root root 932808 Jun 10 2020 postgresql-42.2.14.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] izhangzhihao commented on issue #3484: [SUPPORT] ClassNotFoundException: org.apache.hudi.sink.StreamWriteOperatorCoordinator$Provider
Posted by GitBox <gi...@apache.org>.
izhangzhihao commented on issue #3484:
URL: https://github.com/apache/hudi/issues/3484#issuecomment-899959261
JobManager log
```
2021-08-17 03:07:37,773 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath: /opt/flink/lib/commons-logging-1.2.jar:/opt/flink/lib/flink-connector-postgres-cdc-2.1-SNAPSHOT.jar:/opt/flink/lib/flink-csv-1.13.1.jar:/opt/flink/lib/flink-json-1.13.1.jar:/opt/flink/lib/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-sql-connector-elasticsearch7_2.12-1.13.1.jar:/opt/flink/lib/flink-table-blink_2.12-1.13.1.jar:/opt/flink/lib/flink-table_2.12-1.13.1.jar:/opt/flink/lib/hadoop-common-shaded-SNAPSHOT.jar:/opt/flink/lib/hudi-flink-bundle_2.12-0.10.0-SNAPSHOT.jar:/opt/flink/lib/hudi-utilities-bundle_2.12-0.10.0-SNAPSHOT.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/postgresql-42.2.14.jar:/opt/flink/lib/flink-dist_2.12-1.13.1.jar:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org