You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "deasea (via GitHub)" <gi...@apache.org> on 2023/08/22 08:56:16 UTC
[GitHub] [hudi] deasea opened a new issue, #9500: flink offlinecompact error
deasea opened a new issue, #9500:
URL: https://github.com/apache/hudi/issues/9500
Hello, I encountered an error when I use flink offline compact :
2023-08-21 16:03:22,756 INFO org.apache.flink.runtime.taskmanager.Task [] - compact_task (3/3)#0 (fd49759c8ac941659bb460ca82108333) switched from INITIALIZING to RUNNING.
2023-08-21 16:03:22,757 INFO org.apache.flink.runtime.taskmanager.Task [] - compact_task (2/3)#0 (6caf7347aa5869e2ab11b3d628871ac9) switched from INITIALIZING to RUNNING.
2023-08-21 16:03:23,896 WARN org.apache.flink.runtime.taskmanager.Task [] - compact_task (2/3)#0 (6caf7347aa5869e2ab11b3d628871ac9) switched from RUNNING to FAILED with failure cause: java.lang.RuntimeException: Cannot instantiate class.
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.createInstance(PojoSerializer.java:213)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:413)
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:193)
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
at org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:53)
at org.apache.flink.runtime.io.network.api.serialization.NonSpanningWrapper.readInto(NonSpanningWrapper.java:337)
at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.readNonSpanningRecord(SpillingAdaptiveSpanningRecordDeserializer.java:128)
at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.readNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:103)
at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:93)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:95)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Can not set org.apache.hudi.common.model.CompactionOperation field org.apache.hudi.sink.compact.CompactionPlanEvent.operation to org.apache.hudi.common.model.CompactionOperation
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
at sun.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:81)
at java.lang.reflect.Field.set(Field.java:764)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.initializeFields(PojoSerializer.java:221)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.createInstance(PojoSerializer.java:210)
... 19 more
./bin/flink run \
-c org.apache.hudi.sink.compact.HoodieFlinkCompactor \
lib/hudi-flink1.14-bundle-0.12.0.jar \
--compaction-max-memory 1024 \
--compaction-tasks 3 \
--path hdfs:/.../tableA
* Hudi version : 0.12.0
* Flink version : 0.14.5
* Scala version: 2.12
* Hadoop version : 3.2.1
* Run mode : flink on yarn session
when I run it on my local VM centos, it's ok ; but run product env, it appears error,
please Help me !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9500: flink offlinecompact error
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9500:
URL: https://github.com/apache/hudi/issues/9500#issuecomment-1689127600
Seems like a jar loading issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9500: flink offlinecompact error
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9500:
URL: https://github.com/apache/hudi/issues/9500#issuecomment-1694861170
Did you try to put the bundle jar under flink/lib ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] deasea commented on issue #9500: flink offlinecompact error
Posted by "deasea (via GitHub)" <gi...@apache.org>.
deasea commented on issue #9500:
URL: https://github.com/apache/hudi/issues/9500#issuecomment-1689311969
Can you give me some pointers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org