[GitHub] [spark] artiship edited a comment on pull request #30263: [SPARK-33358][SQL] Return code when command process failed

[GitBox <gi...@apache.org>]

artiship edited a comment on pull request #30263:
URL: https://github.com/apache/spark/pull/30263#issuecomment-722477673


   @HyukjinKwon You can see after the first statement failed, following statement still got executed, and finally let the whole script succeed. 
   
   env
   
   ```
   spark version: 3.0.1
   os: centos 7
   ```
   
   /tmp/tmp.sql
   
   ```sql
   select * from nonexistent_table;
   select 2;
   ```
   
   submit command:
   
   ```sh
   export HADOOP_USER_NAME=my-hadoop-user
   bin/spark-sql  \
   --master yarn \
   --deploy-mode client \
   --queue my.queue.name \
   --conf spark.driver.host=$(hostname -i) \
   --conf spark.app.name=spark-test  \
   --name "spark-test" \
   -f /tmp/tmp.sql 
   ```
   
   execution log:
   
   ```sh
   # bin/spark-sql  \
   > --master yarn \
   > --deploy-mode client \
   > --queue my.queue.name \
   > --conf spark.driver.host=$(hostname -i) \
   > --conf spark.app.name=spark-test  \
   > --name "spark-test" \
   > -f /tmp/tmp.sql
   20/11/06 00:06:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   20/11/06 00:06:20 WARN HiveConf: HiveConf of name hive.spark.client.rpc.server.address.use.ip does not exist
   20/11/06 00:06:20 WARN HiveConf: HiveConf of name hive.spark.client.submit.timeout.interval does not exist
   20/11/06 00:06:20 WARN HiveConf: HiveConf of name hive.enforce.bucketing does not exist
   20/11/06 00:06:20 WARN HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist
   20/11/06 00:06:20 WARN HiveConf: HiveConf of name hive.run.timeout.seconds does not exist
   20/11/06 00:06:20 WARN HiveConf: HiveConf of name hive.support.sql11.reserved.keywords does not exist
   20/11/06 00:06:20 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
   20/11/06 00:06:20 WARN SparkConf: Note that spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone/kubernetes and LOCAL_DIRS in YARN).
   20/11/06 00:06:22 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
   
   Error in query: Table or view not found: nonexistent_table; line 1 pos 14;
   'Project [*]
   +- 'UnresolvedRelation [nonexistent_table]
   
   2
   2
   Time taken: 4.437 seconds, Fetched 1 row(s)
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org