[jira] [Commented] (KYLIN-5126) Build kylin 4.0, spark has not been able to submit to the yarn cluster

["xbchao (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17444238#comment-17444238 ] 

xbchao commented on KYLIN-5126:
-------------------------------

java.io.IOException: OS command error exit with return code: 1, error message: log4j: Using URL [file:/opt/apps/kylin/conf/spark-driver-log4j.properties] for automatic log4j configuration.
log4j: Reading configuration from URL file:/opt/apps/kylin/conf/spark-driver-log4j.properties
log4j: Parsing for [root] with value=[INFO,hdfs].
log4j: Level token is [INFO].
log4j: Category root set to INFO
log4j: Parsing appender named "hdfs".
log4j: Parsing layout options for "hdfs".
log4j: Setting property [conversionPattern] to [%d\{ISO8601} %-5p [%t] %c\{2} : %m%n].
log4j: End of parsing for "hdfs".
log4j: Setting property [hdfsWorkingDir] to [s3://bigdata-kylin/test/kylin_metadata/].
log4j: Setting property [kerberosPrincipal] to [].
log4j: Setting property [logPath] to [s3://bigdata-kylin/test/kylin_metadata/learn_kylin/spark_logs/driver/56de5bb7-265b-433e-9712-8af73b0e0d7c-00/execute_output.json.1636977105389.log].
log4j: Setting property [kerberosEnable] to [false].
log4j: Setting property [kerberosKeytab] to [].
log4j: Setting property [logQueueCapacity] to [5000].
log4j: Setting property [flushInterval] to [5000].
log4j:WARN SparkDriverHdfsLogAppender starting ...
log4j:WARN hdfsWorkingDir -> s3://bigdata-kylin/test/kylin_metadata/
log4j:WARN spark.driver.log4j.appender.hdfs.File -> s3://bigdata-kylin/test/kylin_metadata/learn_kylin/spark_logs/driver/56de5bb7-265b-433e-9712-8af73b0e0d7c-00/execute_output.json.1636977105389.log
log4j:WARN kerberosEnable -> false
log4j:WARN SparkDriverHdfsLogAppender started ...
log4j: Parsed "hdfs" options.
log4j: Parsing for [org.springframework] with value=[WARN].
log4j: Level token is [WARN].
log4j: Category org.springframework set to WARN
log4j: Handling log4j.additivity.org.springframework=[null]
log4j: Parsing for [org.apache.spark] with value=[WARN].
log4j: Level token is [WARN].
log4j: Category org.apache.spark set to WARN
log4j: Handling log4j.additivity.org.apache.spark=[null]
log4j: Parsing for [org.apache.kylin] with value=[DEBUG].
log4j: Level token is [DEBUG].
log4j: Category org.apache.kylin set to DEBUG
log4j: Handling log4j.additivity.org.apache.kylin=[null]
log4j: Finished configuring.
log4j:WARN SparkDriverHdfsLogAppender flush log when shutdown ...
The command is: 
export HADOOP_CONF_DIR=/opt/apps/kylin/hadoop_conf && /usr/lib/spark/bin/spark-submit --class org.apache.kylin.engine.spark.application.SparkEntry  --conf 'spark.hadoop.yarn.timeline-service.enabled=false'  --conf 'spark.master=local'  --conf 'spark.driver.memoryOverhead=2G'  --conf 'spark.sql.shuffle.partitions=4'  --conf 'spark.speculation=false'  --conf 'spark.yarn.queue=default'  --conf 'spark.driver.cores=2'  --conf 'spark.history.fs.logDirectory=hdfs:///kylin/spark-history'  --conf 'spark.driver.memory=4G'  --conf 'spark.driver.extraJavaOptions=-XX:+CrashOnOutOfMemoryError -Dlog4j.configuration=file:/opt/apps/kylin/conf/spark-driver-log4j.properties  -Dkylin.kerberos.enabled=false  -Dkylin.hdfs.working.dir=s3://bigdata-kylin/test/kylin_metadata/  -Dspark.driver.log4j.appender.hdfs.File=s3://bigdata-kylin/test/kylin_metadata/learn_kylin/spark_logs/driver/56de5bb7-265b-433e-9712-8af73b0e0d7c-00/execute_output.json.1636977105389.log  -Dlog4j.debug=true  -Dspark.driver.rest.server.address=ip-local:7070  -Dspark.driver.param.taskId=56de5bb7-265b-433e-9712-8af73b0e0d7c-00  -Dspark.driver.local.logDir=/opt/apps/kylin/logs/spark'  --conf 'spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec'  --conf 'spark.eventLog.enabled=true'  --conf 'spark.shuffle.service.enabled=true'  --conf 'spark.eventLog.dir=hdfs:///kylin/spark-history'  --conf 'spark.yarn.archive=hdfs://ip-emrmaster:8020/kylin/package/spark-libs.jar'  --conf 'spark.sql.adaptive.enabled=false'  --conf 'spark.sql.autoBroadcastJoinThreshold=-1'  --conf 'spark.driver.extraClassPath=/opt/apps/kylin/lib/kylin-parquet-job-4.0.0.jar' --name job_step_56de5bb7-265b-433e-9712-8af73b0e0d7c-00 --jars /opt/apps/kylin/lib/kylin-parquet-job-4.0.0.jar /opt/apps/kylin/lib/kylin-parquet-job-4.0.0.jar -className org.apache.kylin.engine.spark.job.ResourceDetectBeforeCubingJob s3://bigdata-kylin-/test/kylin_metadata/learn_kylin/job_tmp/56de5bb7-265b-433e-9712-8af73b0e0d7c-00_jobId
	at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:98)
	at org.apache.kylin.engine.spark.job.NSparkExecutable.runSparkSubmit(NSparkExecutable.java:282)
	at org.apache.kylin.engine.spark.job.NSparkExecutable.doWork(NSparkExecutable.java:168)
	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:206)
	at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:94)
	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:206)
	at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

> Build kylin 4.0, spark has not been able to submit to the yarn cluster
> ----------------------------------------------------------------------
>
>                 Key: KYLIN-5126
>                 URL: https://issues.apache.org/jira/browse/KYLIN-5126
>             Project: Kylin
>          Issue Type: Bug
>            Reporter: xbchao
>            Priority: Major
>
> When I built kylin 4.0, spark could not be submitted to the yarn cluster. The version used was apache-kylin-4.0.0-bin-spark2, which was deployed in the aws emr server.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)