You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "guoxiaolongzte (JIRA)" <ji...@apache.org> on 2017/07/06 09:22:00 UTC
[jira] [Updated] (SPARK-21325) The shell of 'spark-submit' about
'--jars' and '--fils', jars and files can be placed on local and hdfs.
[ https://issues.apache.org/jira/browse/SPARK-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
guoxiaolongzte updated SPARK-21325:
-----------------------------------
Description:
1.My submit way:
spark-submit --class cn.gxl.TestSql --jars hdfs://nameservice:/gxl/spark-core_2.11-2.3.0-SNAPSHOT.jar,hdfs://nameservice:/gxl/zookeeper-3.4.6.jar --files hdfs://nameservice:/gxl/value1.txt,hdfs://nameservice:/gxl/value2.txt hdfs://nameservice:/gxl/spark_2.0.2_project.jar
2.spark-submit description：
--jars JARS Comma-separated list of local jars to include on the driver and executor classpaths.
--files FILES Comma-separated list of files to be placed in the working directory of each executor. File paths of these files in executors can be accessed via SparkFiles.get(fileName).
3.Problem Description：
jars and files Not only can be placed on local but also can be placed on hdfs.
The description of '' - jars '', that can only be placed on local.This is wrong
The description of '--files' is not clear that can be placed locally or hdfs.This is blurry. Not conducive to the developer to understand and use.
So, this is an optimization feature that deserves to be modified.
> The shell of 'spark-submit' about '--jars' and '--fils', jars and files can be placed on local and hdfs.
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-21325
> URL: https://issues.apache.org/jira/browse/SPARK-21325
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.3.0
> Reporter: guoxiaolongzte
> Priority: Minor
>
> 1.My submit way:
> spark-submit --class cn.gxl.TestSql --jars hdfs://nameservice:/gxl/spark-core_2.11-2.3.0-SNAPSHOT.jar,hdfs://nameservice:/gxl/zookeeper-3.4.6.jar --files hdfs://nameservice:/gxl/value1.txt,hdfs://nameservice:/gxl/value2.txt hdfs://nameservice:/gxl/spark_2.0.2_project.jar
> 2.spark-submit description：
> --jars JARS Comma-separated list of local jars to include on the driver and executor classpaths.
> --files FILES Comma-separated list of files to be placed in the working directory of each executor. File paths of these files in executors can be accessed via SparkFiles.get(fileName).
> 3.Problem Description：
> jars and files Not only can be placed on local but also can be placed on hdfs.
> The description of '' - jars '', that can only be placed on local.This is wrong
> The description of '--files' is not clear that can be placed locally or hdfs.This is blurry. Not conducive to the developer to understand and use.
> So, this is an optimization feature that deserves to be modified.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org