You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "holdenk (JIRA)" <ji...@apache.org> on 2016/10/08 19:05:20 UTC
[jira] [Commented] (SPARK-14212) Add configuration element for
--packages option
[ https://issues.apache.org/jira/browse/SPARK-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558540#comment-15558540 ]
holdenk commented on SPARK-14212:
---------------------------------
So I think this would be a good option to document for Python users, although the root CSV issue has been fixed by including the CSV format inside of Spark its self. You configure a package using `spark.jars.packages` in spark-defaults.conf.
If someone is interested adding this to the documentation `docs/configuration.md` would probably be a good place to document the `spakr.jars.packages` configuration value (and you can see how it can be by looking at the inside of SparkSubmit & SparkSubmitArguments together.
> Add configuration element for --packages option
> -----------------------------------------------
>
> Key: SPARK-14212
> URL: https://issues.apache.org/jira/browse/SPARK-14212
> Project: Spark
> Issue Type: New Feature
> Components: Documentation, PySpark
> Affects Versions: 1.6.1
> Reporter: Russell Jurney
> Labels: config, fun, happy, pants, spark-shell
>
> I use PySpark with the --packages option, for instance to load support for CSV:
> pyspark --packages com.databricks:spark-csv_2.10:1.4.0
> I would like to not have to set this every time at the command line, so a corresponding element for --packages in the configuration file spark-defaults.conf, would be good to have.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org