You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/01/04 12:06:55 UTC
[GitHub] [dolphinscheduler] QlDoors opened a new issue #7794: [Bug] [task-plugin] run spark application failed
QlDoors opened a new issue #7794:
URL: https://github.com/apache/dolphinscheduler/issues/7794
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
when I run spark applition on yarn, I got a filed messasge:
22/01/04 19:24:27 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:804)
at org.apache.spark.sql.SparkSession$Builder.appName(SparkSession.scala:795)
at org.example.WordCount$.main(WordCount.scala:40)
at org.example.WordCount.main(WordCount.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:567)
22/01/04 19:24:27 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;)
22/01/04 19:24:27 ERROR yarn.ApplicationMaster: Uncaught exception:
java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:55)
at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:585)
Caused by: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:804)
at org.apache.spark.sql.SparkSession$Builder.appName(SparkSession.scala:795)
at org.example.WordCount$.main(WordCount.scala:40)
at org.example.WordCount.main(WordCount.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:567)
22/01/04 19:24:27 INFO util.ShutdownHookManager: Shutdown hook called
### What you expected to happen
run successs
### How to reproduce
1. dolphinscheduler version 2.0.1，download bin.jar from https://dolphinscheduler.apache.org/
2. config conf/env/dolphinscheduler_env.sh
```
export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
export HADOOP_CONF_DIR=/etc/hadoop/conf
#export SPARK_HOME1=/opt/cloudera/parcels/CDH/lib/spark
export SPARK_HOME2=/opt/cloudera/parcels/SPARK2/lib/spark2
export PYTHON_HOME=/usr/bin/python
export JAVA_HOME=/usr/java/jdk1.8.0_161
export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
#export FLINK_HOME=/opt/soft/flink
#export DATAX_HOME=/opt/soft/datax
export PATH=$HADOOP_HOME/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$PATH
```
3. cdh version 5.14.4，spark1 version 1.6.0，spark2 version 2.4.0.cloudera2
4. put /opt/cloudera/parcels/SPARK2/lib/spark2/jars/ *.jar to /dolphinscheduler/qinliang/resources/spark-jars/
5. work-flow 'Option Parameters':
```
--jars hdfs:///dolphinscheduler/qinliang/resources/spark-jars/JavaEWAH-0.3.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/RoaringBitmap-0.7.45.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/ST4-4.0.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/aircompressor-0.10.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/antlr-2.7.7.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/antlr-runtime-3.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/antlr4-runtime-4.7.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/aopalliance-repackaged-2.4.0-b34.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/apache-log4j-extras-1.2.17.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/arpack_combined_all-0.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/arrow-format-0.10.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/arrow-memory-0.10.0.jar,hdfs:///dolphinscheduler/qinliang/resou
rces/spark-jars/arrow-vector-0.10.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/avro-1.7.6-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/avro-ipc-1.7.6-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/avro-mapred-1.7.6-cdh5.13.3-hadoop2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/bonecp-0.7.1.RELEASE.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/breeze-macros_2.11-0.13.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/breeze_2.11-0.13.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/calcite-avatica-1.0.0-incubating.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/calcite-core-1.0.0-incubating.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/calcite-linq4j-1.0.0-incubating.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/chill-java-0.9.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/chill_2.11-0.9.3.jar,hdfs:///dolphinscheduler/qinlia
ng/resources/spark-jars/commons-cli-1.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-codec-1.10.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-collections-3.2.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-compiler-3.0.9.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-compress-1.4.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-crypto-1.0.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-dbcp-1.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-httpclient-3.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-io-2.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-lang-2.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-lang3-3.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-logging-1.1.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-math3-3.4.1.jar,h
dfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-net-3.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-pool-1.5.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/compress-lzf-1.0.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/core-1.1.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/datanucleus-api-jdo-3.2.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/datanucleus-core-3.2.10.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/datanucleus-rdbms-3.2.9.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/derby-10.12.1.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/eigenbase-properties-1.1.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/fastutil-6.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/flatbuffers-1.2.0-3f79e055.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/flume-ng-configuration-1.6.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinl
iang/resources/spark-jars/flume-ng-core-1.6.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/flume-ng-sdk-1.6.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/gson-2.2.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-metastore-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-serde-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-shims-0.23-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-shims-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-shims-common-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-shims-scheduler-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hk2-api-2.4.0-b34.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hk2-locator-2.4.0-b34.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hk2-utils-2.4.0-b34.jar,
hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hppc-0.7.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/httpcore-4.4.10.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/ivy-2.4.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-annotations-2.9.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-core-2.9.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-databind-2.9.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-jaxrs-1.8.8.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-module-paranamer-2.9.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-module-scala_2.11-2.9.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-xc-1.8.8.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/janino-3.0.9.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/javassist-3.18.1-GA.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-j
ars/javax.annotation-api-1.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/javax.inject-2.4.0-b34.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/javax.servlet-api-3.1.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/javax.ws.rs-api-2.0.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/javolution-5.5.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jcl-over-slf4j-1.7.25.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jdo-api-3.0.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-client-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-common-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-container-servlet-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-container-servlet-core-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-guava-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-
media-jaxb-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-server-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jetty-6.1.26.cloudera.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jetty-util-6.1.26.cloudera.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jline-2.14.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/joda-time-2.9.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jodd-core-3.5.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/json-20090211.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/json4s-ast_2.11-3.5.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/json4s-core_2.11-3.5.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/json4s-jackson_2.11-3.5.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/json4s-scalap_2.11-3.5.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jta-1.1.jar,hdfs:///dolphinscheduler/qinli
ang/resources/spark-jars/jtransforms-2.4.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jul-to-slf4j-1.7.25.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/kryo-shaded-4.0.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/libfb303-0.9.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/libthrift-0.9.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/logredactor-1.0.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/lz4-java-1.4.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/machinist_2.11-0.6.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/macro-compat_2.11-1.1.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/metrics-core-3.1.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/metrics-graphite-3.1.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/metrics-json-3.1.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/metrics-jvm-3.1.5.jar,hdfs:///dolphins
cheduler/qinliang/resources/spark-jars/mina-core-2.0.0-M5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/minlog-1.3.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/netty-3.9.9.Final.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/netty-all-4.1.17.Final.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/objenesis-2.5.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/opencsv-2.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/orc-core-1.5.5-nohive.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/orc-mapreduce-1.5.5-nohive.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/orc-shims-1.5.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/oro-2.0.8.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/osgi-resource-locator-1.0.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/paranamer-2.8.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-avro-1.5.0-cdh5.13.3.jar
,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-column-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-common-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-encoding-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-format-2.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-hadoop-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-hadoop-bundle-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-jackson-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/py4j-0.10.7.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/pyrolite-4.13.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/scala-compiler-2.11.12.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/scala-library-2.11.12.jar,hdfs:///dolphinscheduler/qinliang/resources/spark
-jars/scala-parser-combinators_2.11-1.1.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/scala-reflect-2.11.12.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/scala-xml_2.11-1.0.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/shapeless_2.11-2.3.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/shims-0.7.45.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/snappy-0.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/snappy-java-1.1.7.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-catalyst_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-core_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-graphx_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-hive-exec_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-hive_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinschedule
r/qinliang/resources/spark-jars/spark-kvstore_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-launcher_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-lineage_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-mllib-local_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-mllib_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-network-common_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-network-shuffle_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-repl_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-sketch_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-sql_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-streaming
-flume-sink_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-streaming-flume_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-streaming_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-tags_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-unsafe_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-yarn_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spire-macros_2.11-0.13.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spire_2.11-0.13.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/stax-api-1.0.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/stream-2.7.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/stringtemplate-3.2.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/univocity-parsers-2.7.3.jar,hdfs:///dolphinsc
heduler/qinliang/resources/spark-jars/validation-api-1.1.0.Final.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/xbean-asm6-shaded-4.8.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/xz-1.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/zstd-jni-1.3.2-2.jar
```
6. run work-flow，yarn logs：
```
22/01/04 19:24:27 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:804)
at org.apache.spark.sql.SparkSession$Builder.appName(SparkSession.scala:795)
at org.example.WordCount$.main(WordCount.scala:40)
at org.example.WordCount.main(WordCount.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:567)
22/01/04 19:24:27 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;)
22/01/04 19:24:27 ERROR yarn.ApplicationMaster: Uncaught exception:
java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:55)
at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:585)
Caused by: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:804)
at org.apache.spark.sql.SparkSession$Builder.appName(SparkSession.scala:795)
at org.example.WordCount$.main(WordCount.scala:40)
at org.example.WordCount.main(WordCount.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:567)
22/01/04 19:24:27 INFO util.ShutdownHookManager: Shutdown hook called
```
### Anything else
_No response_
### Version
2.0.1
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] QlDoors edited a comment on issue #7794: [Bug] [task-plugin] run spark application failed
Posted by GitBox <gi...@apache.org>.
QlDoors edited a comment on issue #7794:
URL: https://github.com/apache/dolphinscheduler/issues/7794#issuecomment-1005357030
解决了，因为dolphinscheduler在执行命令的时候，没有把 conf/env/dolphinscheduler_env.sh 加载到运行环境中造成的，通过创建一个"环境"搞定了，环境配置如下：
```
source /etc/profile
source /home/qinliang/dolphinscheduler/conf/env/dolphinscheduler_env.sh
export HADOOP_USER_NAME=hdfs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] QlDoors closed issue #7794: [Bug] [task-plugin] run spark application failed
Posted by GitBox <gi...@apache.org>.
QlDoors closed issue #7794:
URL: https://github.com/apache/dolphinscheduler/issues/7794
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #7794: [Bug] [task-plugin] run spark application failed
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #7794:
URL: https://github.com/apache/dolphinscheduler/issues/7794#issuecomment-1004753110
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] QlDoors commented on issue #7794: [Bug] [task-plugin] run spark application failed
Posted by GitBox <gi...@apache.org>.
QlDoors commented on issue #7794:
URL: https://github.com/apache/dolphinscheduler/issues/7794#issuecomment-1005357030
解决了，因为dolphinscheduler在执行命令的时候，没有把 conf/env/dolphinscheduler_env.sh 加载到运行环境中造成的，通过创建一个"环境"搞定了，环境配置如下：
`source /etc/profile
source /home/qinliang/dolphinscheduler/conf/env/dolphinscheduler_env.sh
export HADOOP_USER_NAME=hdfs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org