[jira] [Updated] (SPARK-22387) propagate session configs to data source read/write options

["Xiao Li (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-22387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Li updated SPARK-22387:
----------------------------
    Parent Issue: SPARK-15689  (was: SPARK-22386)

> propagate session configs to data source read/write options
> -----------------------------------------------------------
>
>                 Key: SPARK-22387
>                 URL: https://issues.apache.org/jira/browse/SPARK-22387
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Wenchen Fan
>            Assignee: Jiang Xingbo
>            Priority: Major
>             Fix For: 2.3.0
>
>
> This is an open discussion. The general idea is we should allow users to set some common configs in session conf so that they don't need to type them again and again for each data source operations.
> Proposal 1:
> propagate every session config which starts with {{spark.datasource.config.}} to data source options. The downside is, users may only want to set some common configs for a specific data source.
> Proposal 2:
> propagate session config which starts with {{spark.datasource.config.myDataSource.}} only to {{myDataSource}} operations. One downside is, some data source may not have a short name and makes the config key pretty long, e.g. {{spark.datasource.config.com.company.foo.bar.key1}}.
> Proposal 3:
> Introduce a trait `WithSessionConfig` which defines session config key prefix. Then we can pick session configs with this key-prefix and propagate it to this particular data source.
> One another thing also worth to think: sometimes it's really annoying if users have a typo in the config key and spend a lot of time to figure out why things don't work as expected. We should allow data source to validate the given options and throw exception if an option can't be recognized.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org