You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Xiaoxiang Yu (Jira)" <ji...@apache.org> on 2020/07/31 12:26:02 UTC
[jira] [Closed] (KYLIN-4387) Flink cubing merge step failed
[ https://issues.apache.org/jira/browse/KYLIN-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiaoxiang Yu closed KYLIN-4387.
-------------------------------
Resolved in release 3.1.0 (2020-07-03)
> Flink cubing merge step failed
> ------------------------------
>
> Key: KYLIN-4387
> URL: https://issues.apache.org/jira/browse/KYLIN-4387
> Project: Kylin
> Issue Type: Bug
> Components: Flink Engine
> Reporter: Harvey Yue
> Assignee: Harvey Yue
> Priority: Major
> Fix For: v3.1.0
>
>
> When job running this step face following issue, I guess which have two class loader to loading the class *InstantiatorStrategy* when flink execute the reduceGroup operator
> Workaround：using ByteArray instead of Text in reduceGroup operator
> The kylin binary including the flink core jar is the root cause, we need to change the flink related dependence scope to provided.
>
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: error execute org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. (JobID: 7a40a92561cfa972e3c5b975db993fce)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593)
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
> at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
> Caused by: java.lang.RuntimeException: error execute org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. (JobID: 7a40a92561cfa972e3c5b975db993fce)
> at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.FlinkEntry.main(FlinkEntry.java:46)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
> ... 12 more
> Caused by: org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 7a40a92561cfa972e3c5b975db993fce)
> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326)
> at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
> at org.apache.kylin.engine.flink.FlinkCubingMerge.execute(FlinkCubingMerge.java:235)
> at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> ... 18 more
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259)
> ... 23 more
> Caused by: java.lang.LinkageError: loader constraint violation: when resolving method "com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(*Lorg/objenesis/strategy/InstantiatorStrategy*;)V" the class loader (instance of org/apache/flink/util/*ChildFirstClassLoader*) of the current class, org/apache/flink/api/java/typeutils/runtime/WritableComparator, and the class loader (instance of sun/misc/Launcher$*AppClassLoader*) for the method's defining class, com/esotericsoftware/kryo/Kryo$DefaultInstantiatorStrategy, have different Class objects for the type org/objenesis/strategy/InstantiatorStrategy used in the signature
> at org.apache.flink.api.java.typeutils.runtime.WritableComparator.checkKryoInitialized(WritableComparator.java:174)
> at org.apache.flink.api.java.typeutils.runtime.WritableComparator.setReference(WritableComparator.java:67)
> at org.apache.flink.api.java.typeutils.runtime.WritableComparator.setReference(WritableComparator.java:38)
> at org.apache.flink.api.java.typeutils.runtime.TupleComparator.setReference(TupleComparator.java:73)
> at org.apache.flink.api.java.typeutils.runtime.TupleComparator.setReference(TupleComparator.java:30)
> at org.apache.flink.runtime.operators.ReduceDriver.run(ReduceDriver.java:137)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:504)
> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:369)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
> at java.lang.Thread.run(Thread.java:748)
> The command is:
> export HADOOP_CONF_DIR=/etc/hadoop/conf && export HADOOP_CLASSPATH=/etc/hadoop && /opt/flink/bin/flink run -m yarn-cluster -ytm 4G -yjm 2G -yD taskmanager.memory.preallocate false -ys 1 -c org.apache.kylin.common.util.FlinkEntry -p 20 /data/kylin_poc/lib/kylin-job-3.0.1.jar -className org.apache.kylin.engine.flink.FlinkCubingMerge -output hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-68cac798-dbd6-ae5a-4fac-7229a23aacdc/dc_mileage_cube/cuboid/ -input hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-60802d42-d736-06ea-566e-0f5dc0bf9964/dc_mileage_cube/cuboid/,hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-4f4b2473-6b44-3831-2ff2-035186e94a7f/dc_mileage_cube/cuboid/ -enableObjectReuse false -segmentId 49cb1d56-c6ee-fe73-c431-f2891faa1efb -metaUrl kylin_metadata_poc@hdfs,path=hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-68cac798-dbd6-ae5a-4fac-7229a23aacdc/dc_mileage_cube/metadata -cubename dc_mileage_cube
--
This message was sent by Atlassian Jira
(v8.3.4#803005)