You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by GitBox <gi...@apache.org> on 2020/10/20 01:55:03 UTC
[GitHub] [kylin] chenjie-sau removed a comment on pull request #1451: KYLIN-4736 Upgrade Flink version to 1.11.1
chenjie-sau removed a comment on pull request #1451:
URL: https://github.com/apache/kylin/pull/1451#issuecomment-712538067
> I tested this branch in my local env. I use the $KYLIN_HOME/bin/download-flink.sh download Flink 1.11.1. Then I start kylin and submit a Flink build job. But there is a error in step `Build cube with flink`. The error information is as follows:
>
> ```
> ------------------------------------------------------------
> The program finished with the following exception:
>
> org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: error execute org.apache.kylin.engine.flink.FlinkCubingByLayer. Root cause: Filesystem closed
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
> at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:699)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:232)
> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916)
> at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
> Caused by: java.lang.RuntimeException: error execute org.apache.kylin.engine.flink.FlinkCubingByLayer. Root cause: Filesystem closed
> at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.FlinkEntry.main(FlinkEntry.java:46)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
> ... 11 more
> Caused by: java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:476)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1683)
> at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1714)
> at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1711)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1726)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1718)
> at org.apache.kylin.engine.mr.JobBuilderSupport.scanFiles(JobBuilderSupport.java:495)
> at org.apache.kylin.engine.mr.JobBuilderSupport.getFileSize(JobBuilderSupport.java:510)
> at org.apache.kylin.engine.flink.FlinkCubingByLayer.execute(FlinkCubingByLayer.java:212)
> at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> ... 17 more
> The command is:
> export HADOOP_CONF_DIR=/etc/hadoop/conf && export HADOOP_CLASSPATH=/etc/hadoop && /root/zhangyaqian/tar/apache-kylin-3.1.2-SNAPSHOT-bin/flink/bin/flink run -m yarn-cluster -yjm 2G -ys 1 -ytm 4G -ynm Build-Cube-with-Flink:kylin_sales_cube_clone[20120101000000_20120201000000] -c org.apache.kylin.common.util.FlinkEntry -p 1 /root/zhangyaqian/tar/apache-kylin-3.1.2-SNAPSHOT-bin/lib/kylin-job-3.1.2-SNAPSHOT.jar -className org.apache.kylin.engine.flink.FlinkCubingByLayer -hiveTable default.kylin_intermediate_kylin_sales_cube_clone_3ca685b1_049d_4377_9c9f_1698e0cbcbc6 -output hdfs://cdh-master:8020/kylin/yaqian/kylin_zyq/kylin-44637f80-4271-4dcd-0fb2-f673a418b80e/kylin_sales_cube_clone/cuboid/ -input hdfs://cdh-master:8020/kylin/yaqian/kylin_zyq/kylin-44637f80-4271-4dcd-0fb2-f673a418b80e/kylin_intermediate_kylin_sales_cube_clone_3ca685b1_049d_4377_9c9f_1698e0cbcbc6 -enableObjectReuse false -segmentId 3ca685b1-049d-4377-9c9f-1698e0cbcbc6 -metaUrl kylin_zyq@hdfs,path=hdfs://cdh-master:
8020/kylin/yaqian/kylin_zyq/kylin-44637f80-4271-4dcd-0fb2-f673a418b80e/kylin_sales_cube_clone/metadata -cubename kylin_sales_cube_clone
> at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:96)
> at org.apache.kylin.engine.flink.FlinkExecutable.lambda$doWork$1(FlinkExecutable.java:246)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-10-16 14:23:39,537 INFO [Scheduler 737950975 Job 44637f80-4271-4dcd-0fb2-f673a418b80e-123] execution.ExecutableManager:485 : job id:44637f80-4271-4dcd-0fb2-f673a418b80e-06 from RUNNING to ERROR
> 2020-10-16 14:23:39,553 INFO [Scheduler 737950975 Job 44637f80-4271-4dcd-0fb2-f673a418b80e-123] execution.ExecutableManager:485 : job id:44637f80-4271-4dcd-0fb2-f673a418b80e from RUNNING to ERROR
> ```
>
> @chenjie-sau Do you know the reason for this error?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org