You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by MaxGekk <gi...@git.apache.org> on 2018/09/19 19:46:19 UTC
[GitHub] spark pull request #22474: [SPARK-25425][SQL][BACKPORT-2.4] Extra options sh...
GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/22474
[SPARK-25425][SQL][BACKPORT-2.4] Extra options should override session options in DataSource V2
## What changes were proposed in this pull request?
In the PR, I propose overriding session options by extra options in DataSource V2. Extra options are more specific and set via `.option()`, and should overwrite more generic session options.
## How was this patch tested?
Added tests for read and write paths.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 session-options-2.4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22474.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22474
----
commit ac704568591d2c505b396b5c93138ea8132e3cc4
Author: Maxim Gekk <ma...@...>
Date: 2018-09-16T00:24:11Z
[SPARK-25425][SQL] Extra options should override session options in DataSource V2
In the PR, I propose overriding session options by extra options in DataSource V2. Extra options are more specific and set via `.option()`, and should overwrite more generic session options. Entries from seconds map overwrites entries with the same key from the first map, for example:
```Scala
scala> Map("option" -> false) ++ Map("option" -> true)
res0: scala.collection.immutable.Map[String,Boolean] = Map(option -> true)
```
Added a test for checking which option is propagated to a data source in `load()`.
Closes #22413 from MaxGekk/session-options.
Lead-authored-by: Maxim Gekk <ma...@databricks.com>
Co-authored-by: Dongjoon Hyun <do...@apache.org>
Co-authored-by: Maxim Gekk <ma...@gmail.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
commit bf3a52c1240a14f89949f8b9dcb554090f803250
Author: Maxim Gekk <ma...@...>
Date: 2018-09-19T19:40:25Z
Adding missing import after merge to 2.4
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22474: [SPARK-25425][SQL][BACKPORT-2.4] Extra options should ov...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22474
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22474: [SPARK-25425][SQL][BACKPORT-2.4] Extra options should ov...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22474
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22474: [SPARK-25425][SQL][BACKPORT-2.4] Extra options should ov...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22474
Since it's merged, could you close this PR?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22474: [SPARK-25425][SQL][BACKPORT-2.4] Extra options sh...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk closed the pull request at:
https://github.com/apache/spark/pull/22474
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22474: [SPARK-25425][SQL][BACKPORT-2.4] Extra options should ov...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22474
**[Test build #96282 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96282/testReport)** for PR 22474 at commit [`bf3a52c`](https://github.com/apache/spark/commit/bf3a52c1240a14f89949f8b9dcb554090f803250).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22474: [SPARK-25425][SQL][BACKPORT-2.4] Extra options should ov...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22474
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96282/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22474: [SPARK-25425][SQL][BACKPORT-2.4] Extra options should ov...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22474
**[Test build #96282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96282/testReport)** for PR 22474 at commit [`bf3a52c`](https://github.com/apache/spark/commit/bf3a52c1240a14f89949f8b9dcb554090f803250).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22474: [SPARK-25425][SQL][BACKPORT-2.4] Extra options should ov...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22474
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22474: [SPARK-25425][SQL][BACKPORT-2.4] Extra options should ov...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22474
Thank you, @MaxGekk . Merged to `branch-2.4`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org