You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/25 00:06:28 UTC
[GitHub] [spark] huaxingao opened a new pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
huaxingao opened a new pull request #29535:
URL: https://github.com/apache/spark/pull/29535
### What changes were proposed in this pull request?
pass specified options in DataFrameReader.table to JDBCTableCatalog.loadTable
### Why are the changes needed?
Currently, `DataFrameReader.table` ignores the specified options. The options specified like the following are lost.
```
val df = spark.read
.option("partitionColumn", "id")
.option("lowerBound", "0")
.option("upperBound", "3")
.option("numPartitions", "2")
.table("h2.test.people")
```
We need to make `DataFrameReader.table` take the specified options.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually test for now. Will add a test after V2 JDBC read is implemented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682079723
**[Test build #127962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127962/testReport)** for PR 29535 at commit [`dcc88ca`](https://github.com/apache/spark/commit/dcc88ca041201de9ef454d2c9d5f46c0c2e6b114).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r479825952
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -544,6 +545,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
case None => Nil
case Some(null) => Nil
case Some(any) => any :: Nil
+ case map: CaseInsensitiveStringMap =>
+ truncatedString(map.entrySet().toArray(), "[", ", ", "]", maxFields) :: Nil
Review comment:
Seems we can't have a ```UnresolvedRelation``` with a non empty ```CaseInsensitiveStringMap``` in explain? The only way to create a ```UnresolvedRelation``` with a non empty ```CaseInsensitiveStringMap``` is using ```DataFrameReader.table```, but this is resolved to a ```V2Relation```.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682172886
**[Test build #127965 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127965/testReport)** for PR 29535 at commit [`f321560`](https://github.com/apache/spark/commit/f321560c199dde2e00171536a1d589e9a839a1a9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681687341
**[Test build #127948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127948/testReport)** for PR 29535 at commit [`d56f92d`](https://github.com/apache/spark/commit/d56f92d9b0722ac9ebec471e2bb147141e66f7dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682062507
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127961/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682043554
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127959/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682062464
**[Test build #127961 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127961/testReport)** for PR 29535 at commit [`a50bf2f`](https://github.com/apache/spark/commit/a50bf2f83afca1929c989734c68b795c225c3a65).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-679702212
cc @MaxGekk @cloud-fan @viirya
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681189913
**[Test build #127943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127943/testReport)** for PR 29535 at commit [`0bc7698`](https://github.com/apache/spark/commit/0bc76982c4873b184096fce33981321dec751d84).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r714488634
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -40,9 +41,12 @@ class UnresolvedException[TreeType <: TreeNode[_]](tree: TreeType, function: Str
* Holds the name of a relation that has yet to be looked up in a catalog.
*
* @param multipartIdentifier table name
+ * @param options options to scan this relation. Only applicable to v2 table scan.
Review comment:
Oh, I created a PR without seeing your comment. Thanks @huaxingao.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r714469347
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala
##########
@@ -186,4 +187,21 @@ class DataSourceV2DataFrameSuite
assert(e3.getMessage.contains(s"Cannot use interval type in the table schema."))
}
}
+
+ test("options to scan v2 table should be passed to DataSourceV2Relation") {
+ val t1 = "testcat.ns1.ns2.tbl"
+ withTable(t1) {
+ val df1 = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data")
+ df1.write.saveAsTable(t1)
+
+ val optionName = "fakeOption"
+ val df2 = spark.read
+ .option(optionName, false)
+ .table(t1)
Review comment:
so for doubly sure, what happen if some options are already set in this table? e.g.)
```scala
sql("CREATE TABLE tbl(a int) USING jdbc OPTIONS(a='b')")
spark.option("a", "c").table("tbl")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681689063
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682076486
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683598681
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683622578
**[Test build #128084 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128084/testReport)** for PR 29535 at commit [`a41c06a`](https://github.com/apache/spark/commit/a41c06af4ba4edf73a267763bd26f8fd73b13b97).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683623187
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-679686309
**[Test build #127860 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127860/testReport)** for PR 29535 at commit [`b54cc55`](https://github.com/apache/spark/commit/b54cc55c85055ced0582a0e0c513c2e2c1fee504).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r477048504
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -822,6 +821,8 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
*/
def table(tableName: String): DataFrame = {
assertNoSpecifiedSchema("table")
+ for ((k, v) <- this.extraOptions)
+ sparkSession.conf.set(k, v)
Review comment:
Currently, we already pass options from `DataFrameReader` to `DataSourceV2Relation` in `load` API. `DataFrameReader.table` misses this part.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r479393860
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -544,6 +545,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
case None => Nil
case Some(null) => Nil
case Some(any) => any :: Nil
+ case map: CaseInsensitiveStringMap =>
+ truncatedString(map.entrySet().toArray(), "[", ", ", "]", maxFields) :: Nil
Review comment:
we can use `asCaseSensitiveMap` call.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r478216522
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -928,12 +928,15 @@ class Analyzer(
/**
* Performs the lookup of DataSourceV2 Tables from v2 catalog.
*/
- private def lookupV2Relation(identifier: Seq[String]): Option[DataSourceV2Relation] =
+ private def lookupV2Relation(
+ identifier: Seq[String],
+ options: CaseInsensitiveStringMap = CaseInsensitiveStringMap.empty())
Review comment:
do we really need the default value?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682367419
**[Test build #127988 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127988/testReport)** for PR 29535 at commit [`12d90bd`](https://github.com/apache/spark/commit/12d90bd43630fc38c6bf7860dd6ded17376862d7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r714476408
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -40,9 +41,12 @@ class UnresolvedException[TreeType <: TreeNode[_]](tree: TreeType, function: Str
* Holds the name of a relation that has yet to be looked up in a catalog.
*
* @param multipartIdentifier table name
+ * @param options options to scan this relation. Only applicable to v2 table scan.
Review comment:
okay, I just noticed https://github.com/apache/spark/commit/5e825482d70e13a8cb16f1fbdac8139710482d17 added the merging behaviour.
Okay, maybe we should fix the comments here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-679427549
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681696781
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127948/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683594715
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681183024
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r479422041
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -544,6 +545,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
case None => Nil
case Some(null) => Nil
case Some(any) => any :: Nil
+ case map: CaseInsensitiveStringMap =>
+ truncatedString(map.entrySet().toArray(), "[", ", ", "]", maxFields) :: Nil
Review comment:
I tried this:
```
scala> val options = Map(
| "Key1" -> "Value1",
| "key2" -> "vaue2"
| )
options: scala.collection.immutable.Map[String,String] = Map(Key1 -> Value1, key2 -> vaue2)
scala> val map = new CaseInsensitiveStringMap(options.asJava)
map: org.apache.spark.sql.util.CaseInsensitiveStringMap = org.apache.spark.sql.util.CaseInsensitiveStringMap@9f174175
scala> map.asCaseSensitiveMap().entrySet().toArray()
res3: Array[Object] = Array(Key1=Value1, key2=vaue2)
```
I used `case map: CaseInsensitiveStringMap` for pattern matching, I guess we are probably OK?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682172886
**[Test build #127965 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127965/testReport)** for PR 29535 at commit [`f321560`](https://github.com/apache/spark/commit/f321560c199dde2e00171536a1d589e9a839a1a9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r478586635
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -40,7 +41,6 @@ import org.apache.spark.sql.execution.datasources.csv._
import org.apache.spark.sql.execution.datasources.jdbc._
import org.apache.spark.sql.execution.datasources.json.TextInputJsonDataSource
import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation, DataSourceV2Utils}
-import org.apache.spark.sql.internal.SQLConf
Review comment:
Oh, @huaxingao, don't remove this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681689063
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681696690
**[Test build #127948 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127948/testReport)** for PR 29535 at commit [`d56f92d`](https://github.com/apache/spark/commit/d56f92d9b0722ac9ebec471e2bb147141e66f7dc).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683603230
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r479926921
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -544,6 +545,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
case None => Nil
case Some(null) => Nil
case Some(any) => any :: Nil
+ case map: CaseInsensitiveStringMap =>
+ truncatedString(map.entrySet().toArray(), "[", ", ", "]", maxFields) :: Nil
Review comment:
added a test. Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683768073
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-925503778
it's table properties vs scan options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-679691360
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682060089
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681190302
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682068191
@huaxingao You may need to sync with master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681461215
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681181596
**[Test build #127942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127942/testReport)** for PR 29535 at commit [`05b55cb`](https://github.com/apache/spark/commit/05b55cbb5107597fc85687c355ecb0b983605f8e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r479907456
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -544,6 +545,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
case None => Nil
case Some(null) => Nil
case Some(any) => any :: Nil
+ case map: CaseInsensitiveStringMap =>
+ truncatedString(map.entrySet().toArray(), "[", ", ", "]", maxFields) :: Nil
Review comment:
`df.explain(true)` will explain the parsed plan, which contains `UnresolvedRelation`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682059385
**[Test build #127961 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127961/testReport)** for PR 29535 at commit [`a50bf2f`](https://github.com/apache/spark/commit/a50bf2f83afca1929c989734c68b795c225c3a65).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682082969
**[Test build #127962 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127962/testReport)** for PR 29535 at commit [`dcc88ca`](https://github.com/apache/spark/commit/dcc88ca041201de9ef454d2c9d5f46c0c2e6b114).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682483753
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682170097
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683622578
**[Test build #128084 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128084/testReport)** for PR 29535 at commit [`a41c06a`](https://github.com/apache/spark/commit/a41c06af4ba4edf73a267763bd26f8fd73b13b97).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681480921
@cloud-fan @viirya Could you please check one more time? Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-925500304
wait, I get confused here. We already defined a table with options. How does it work with the newly set options? are they merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-925504281
this creates a myth that setting `options` will overwrite table properties. see also https://github.com/apache/spark/pull/34072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r479426100
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -544,6 +545,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
case None => Nil
case Some(null) => Nil
case Some(any) => any :: Nil
+ case map: CaseInsensitiveStringMap =>
+ truncatedString(map.entrySet().toArray(), "[", ", ", "]", maxFields) :: Nil
Review comment:
Sure. Will do. Have a good night!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681693653
Fixed. Thank you! @cloud-fan @viirya
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681190302
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683603363
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128078/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-679427286
**[Test build #127860 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127860/testReport)** for PR 29535 at commit [`b54cc55`](https://github.com/apache/spark/commit/b54cc55c85055ced0582a0e0c513c2e2c1fee504).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r478180602
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -822,7 +822,9 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
*/
def table(tableName: String): DataFrame = {
assertNoSpecifiedSchema("table")
- sparkSession.table(tableName)
+ val multipartIdentifier =
+ sparkSession.sessionState.sqlParser.parseMultipartIdentifier(tableName)
+ Dataset.ofRows(sparkSession, UnresolvedRelation(multipartIdentifier, extraOptions))
Review comment:
we should create `CaseInsensitiveStringMap` here:
```
new CaseInsensitiveStringMap(extraOptions.toMap)
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -822,7 +822,9 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
*/
def table(tableName: String): DataFrame = {
assertNoSpecifiedSchema("table")
- sparkSession.table(tableName)
+ val multipartIdentifier =
+ sparkSession.sessionState.sqlParser.parseMultipartIdentifier(tableName)
+ Dataset.ofRows(sparkSession, UnresolvedRelation(multipartIdentifier, extraOptions))
Review comment:
we should create `CaseInsensitiveStringMap` here:
```
new CaseInsensitiveStringMap(extraOptions.toMap.asJava)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r479413335
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -544,6 +545,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
case None => Nil
case Some(null) => Nil
case Some(any) => any :: Nil
+ case map: CaseInsensitiveStringMap =>
+ truncatedString(map.entrySet().toArray(), "[", ", ", "]", maxFields) :: Nil
Review comment:
I tried the following
```
scala> Map("a" -> 1).asJava.entrySet.toArray.head
res1: Object = scala.collection.convert.Wrappers$MapWrapper$$anon$1$$anon$5$$anon$6@60
```
I think it's safer to generate the string `key=value` manually.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682365208
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683598228
**[Test build #128078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128078/testReport)** for PR 29535 at commit [`a41c06a`](https://github.com/apache/spark/commit/a41c06af4ba4edf73a267763bd26f8fd73b13b97).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683603335
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682039608
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682365208
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682082991
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-679691360
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682247995
**[Test build #127965 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127965/testReport)** for PR 29535 at commit [`f321560`](https://github.com/apache/spark/commit/f321560c199dde2e00171536a1d589e9a839a1a9).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683872215
Thank you very much! @cloud-fan @viirya
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682076486
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r477045489
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -822,6 +821,8 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
*/
def table(tableName: String): DataFrame = {
assertNoSpecifiedSchema("table")
+ for ((k, v) <- this.extraOptions)
+ sparkSession.conf.set(k, v)
Review comment:
1. We don't need to change `SparkSession.table`. We can create `UnresolvedRelation` in `DataFrameReader.load` directly.
2. This is options to scan the v2 table, not v2 table properties. We don't need to change `TableCatalog.loadTable`, but pass the scan options to `Table.newScanBuilder`. `DataSourceV2Relation` already takes the scan options, we just need to create `DataSourceV2Relation` with scan options in `ResolveTables` and `ResolveRelations`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681183029
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127942/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-925504126
So `UnresolvedReleation` is shared for both cases but conditionally use the `UnresolvedReleation.options` only for Scan? that's very confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681181901
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683621225
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-925509816
> this creates a myth that setting options will overwrite table properties.
This is expected. Per-scan options have higher priority than table properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682248198
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682039608
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r478216647
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -1016,7 +1019,10 @@ class Analyzer(
// 1) If the resolved catalog is not session catalog, return None.
// 2) If a relation is not found in the catalog, return None.
// 3) If a v1 table is found, create a v1 relation. Otherwise, create a v2 relation.
- private def lookupRelation(identifier: Seq[String]): Option[LogicalPlan] = {
+ private def lookupRelation(
+ identifier: Seq[String],
+ options: CaseInsensitiveStringMap = CaseInsensitiveStringMap.empty())
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681189913
**[Test build #127943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127943/testReport)** for PR 29535 at commit [`0bc7698`](https://github.com/apache/spark/commit/0bc76982c4873b184096fce33981321dec751d84).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r478179037
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -42,7 +42,9 @@ class UnresolvedException[TreeType <: TreeNode[_]](tree: TreeType, function: Str
* @param multipartIdentifier table name
*/
case class UnresolvedRelation(
- multipartIdentifier: Seq[String]) extends LeafNode with NamedRelation {
+ multipartIdentifier: Seq[String],
+ options: CaseInsensitiveMap[String] = CaseInsensitiveMap[String](Map.empty))
Review comment:
can we use `CaseInsensitiveStringMap` since it's only for v2?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682043517
**[Test build #127959 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127959/testReport)** for PR 29535 at commit [`a50bf2f`](https://github.com/apache/spark/commit/a50bf2f83afca1929c989734c68b795c225c3a65).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r478586635
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -40,7 +41,6 @@ import org.apache.spark.sql.execution.datasources.csv._
import org.apache.spark.sql.execution.datasources.jdbc._
import org.apache.spark.sql.execution.datasources.json.TextInputJsonDataSource
import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation, DataSourceV2Utils}
-import org.apache.spark.sql.internal.SQLConf
Review comment:
Oh, @huaxingao, don't remove this. Other change uses `SQLConf`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r478217017
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -40,9 +41,12 @@ class UnresolvedException[TreeType <: TreeNode[_]](tree: TreeType, function: Str
* Holds the name of a relation that has yet to be looked up in a catalog.
*
* @param multipartIdentifier table name
+ * @param options options for this relation
Review comment:
options to scan this relation. Only applicable to v2 table scan.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682063911
it will be tested in the JDBC v2 PR, but yea, better to have a UT here with a fake v2 source.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682082998
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127962/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682097656
@huaxingao can you add a test in `DataSourceV2DataFrameSuite`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r476942712
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -822,6 +821,8 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
*/
def table(tableName: String): DataFrame = {
assertNoSpecifiedSchema("table")
+ for ((k, v) <- this.extraOptions)
+ sparkSession.conf.set(k, v)
Review comment:
@cloud-fan Thanks for your comment.
I took a look at the code. I think I need to change the following:
<img width="896" alt="Screen Shot 2020-08-25 at 6 03 49 PM" src="https://user-images.githubusercontent.com/13592258/91242868-5aa37e80-e6fd-11ea-8498-2554d5c9367a.png">
<img width="900" alt="Screen Shot 2020-08-25 at 6 02 38 PM" src="https://user-images.githubusercontent.com/13592258/91242806-38116580-e6fd-11ea-8057-91ee3c3ef58e.png">
<img width="859" alt="Screen Shot 2020-08-25 at 6 02 50 PM" src="https://user-images.githubusercontent.com/13592258/91242813-3a73bf80-e6fd-11ea-9b67-5196f4597f73.png">
I need to modify lots of files because of the above changes. Could you please check if this looks OK to you before I continue? Thank you very much!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682482948
**[Test build #127988 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127988/testReport)** for PR 29535 at commit [`12d90bd`](https://github.com/apache/spark/commit/12d90bd43630fc38c6bf7860dd6ded17376862d7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r714476408
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -40,9 +41,12 @@ class UnresolvedException[TreeType <: TreeNode[_]](tree: TreeType, function: Str
* Holds the name of a relation that has yet to be looked up in a catalog.
*
* @param multipartIdentifier table name
+ * @param options options to scan this relation. Only applicable to v2 table scan.
Review comment:
okay, I just noticed https://github.com/apache/spark/commit/5e825482d70e13a8cb16f1fbdac8139710482d17 added the merging behaviour for V1.
Okay, maybe we should fix the comments here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r714488634
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -40,9 +41,12 @@ class UnresolvedException[TreeType <: TreeNode[_]](tree: TreeType, function: Str
* Holds the name of a relation that has yet to be looked up in a catalog.
*
* @param multipartIdentifier table name
+ * @param options options to scan this relation. Only applicable to v2 table scan.
Review comment:
Oh, I create a PR without seeing your comment. Thanks @huaxingao.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683598681
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682057426
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682082991
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681181901
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683595474
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682248202
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127965/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681461215
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683594715
**[Test build #128077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128077/testReport)** for PR 29535 at commit [`b371912`](https://github.com/apache/spark/commit/b3719120b396386a22e1e9d41f4141d6ae846e8f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683623187
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681687341
**[Test build #127948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127948/testReport)** for PR 29535 at commit [`d56f92d`](https://github.com/apache/spark/commit/d56f92d9b0722ac9ebec471e2bb147141e66f7dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r478139414
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -42,7 +42,9 @@ class UnresolvedException[TreeType <: TreeNode[_]](tree: TreeType, function: Str
* @param multipartIdentifier table name
Review comment:
Add `options` to param doc?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682335089
@huaxingao good point. I think we should fix `UnresolvedRelation` to have a better format, by overriding `verboseStringWithOperatorId`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682483753
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682079723
**[Test build #127962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127962/testReport)** for PR 29535 at commit [`dcc88ca`](https://github.com/apache/spark/commit/dcc88ca041201de9ef454d2c9d5f46c0c2e6b114).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682317297
We need to fix `explain-aqe.sql`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r477045489
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -822,6 +821,8 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
*/
def table(tableName: String): DataFrame = {
assertNoSpecifiedSchema("table")
+ for ((k, v) <- this.extraOptions)
+ sparkSession.conf.set(k, v)
Review comment:
1. We don't need to change `SparkSession.table`. We can create `UnresolvedRelation` in `DataFrameReader.table` directly.
2. This is options to scan the v2 table, not v2 table properties. We don't need to change `TableCatalog.loadTable`, but pass the scan options to `Table.newScanBuilder`. `DataSourceV2Relation` already takes the scan options, we just need to create `DataSourceV2Relation` with scan options in `ResolveTables` and `ResolveRelations`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r714487810
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -40,9 +41,12 @@ class UnresolvedException[TreeType <: TreeNode[_]](tree: TreeType, function: Str
* Holds the name of a relation that has yet to be looked up in a catalog.
*
* @param multipartIdentifier table name
+ * @param options options to scan this relation. Only applicable to v2 table scan.
Review comment:
opened a PR https://github.com/apache/spark/pull/34075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683773652
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683768073
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681181596
**[Test build #127942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127942/testReport)** for PR 29535 at commit [`05b55cb`](https://github.com/apache/spark/commit/05b55cbb5107597fc85687c355ecb0b983605f8e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682038993
**[Test build #127959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127959/testReport)** for PR 29535 at commit [`a50bf2f`](https://github.com/apache/spark/commit/a50bf2f83afca1929c989734c68b795c225c3a65).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r478181061
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -822,7 +822,9 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
*/
def table(tableName: String): DataFrame = {
assertNoSpecifiedSchema("table")
- sparkSession.table(tableName)
+ val multipartIdentifier =
+ sparkSession.sessionState.sqlParser.parseMultipartIdentifier(tableName)
+ Dataset.ofRows(sparkSession, UnresolvedRelation(multipartIdentifier, extraOptions))
Review comment:
`toMap` is required to get the original map (keys are not lowercased)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681696753
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682248198
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r476194909
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -822,6 +821,8 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
*/
def table(tableName: String): DataFrame = {
assertNoSpecifiedSchema("table")
+ for ((k, v) <- this.extraOptions)
+ sparkSession.conf.set(k, v)
Review comment:
We need to clearly define the behavior here.
I think the options don't make sense if we read temp view, view, or v1 tables in the session catalog. For v1 table, they are created with `CREATE TABLE ... USING ... OPTIONS ...`, so the options are already there and it's confusing to overwrite them per scan.
We do need to keep the scan options when reading v2 tables, but I don't think session conf is the right channel. I think we should put the options in `UnresolvedRelation` and apply these options when we resolve `UnresolvedRelation` to v2 relations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682170097
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683603335
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682367419
**[Test build #127988 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127988/testReport)** for PR 29535 at commit [`12d90bd`](https://github.com/apache/spark/commit/12d90bd43630fc38c6bf7860dd6ded17376862d7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r714485710
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -40,9 +41,12 @@ class UnresolvedException[TreeType <: TreeNode[_]](tree: TreeType, function: Str
* Holds the name of a relation that has yet to be looked up in a catalog.
*
* @param multipartIdentifier table name
+ * @param options options to scan this relation. Only applicable to v2 table scan.
Review comment:
I will have a followup to fix the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681456113
**[Test build #127943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127943/testReport)** for PR 29535 at commit [`0bc7698`](https://github.com/apache/spark/commit/0bc76982c4873b184096fce33981321dec751d84).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681696753
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682062496
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682060089
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r479200244
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -544,6 +545,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
case None => Nil
case Some(null) => Nil
case Some(any) => any :: Nil
+ case map: CaseInsensitiveStringMap =>
+ truncatedString(map.entrySet().toArray(), "[", ", ", "]", maxFields) :: Nil
Review comment:
and we should at least display the original map (keys are not lowercased)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683767007
**[Test build #128084 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128084/testReport)** for PR 29535 at commit [`a41c06a`](https://github.com/apache/spark/commit/a41c06af4ba4edf73a267763bd26f8fd73b13b97).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682370204
I didn't run the whole sql test suites on my local. I will just take chance and test this on Jenkins. If the change causes other problems, I will fix tomorrow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682333545
For this explain formatted
```
EXPLAIN FORMATTED CREATE VIEW explain_view AS SELECT key, val FROM explain_temp1
```
Now it has
```
......
(3) UnresolvedRelation
Arguments: [explain_temp1], org.apache.spark.sql.util.CaseInsensitiveStringMap@1f
......
```
Do we just need to change the expected output to let the test pass? Or we need to fix the code to make the explain string to have more meaningful info than `org.apache.spark.sql.util.CaseInsensitiveStringMap@1f`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r479198876
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -544,6 +545,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
case None => Nil
case Some(null) => Nil
case Some(any) => any :: Nil
+ case map: CaseInsensitiveStringMap =>
+ truncatedString(map.entrySet().toArray(), "[", ", ", "]", maxFields) :: Nil
Review comment:
are you sure this returns meaningful string?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681183010
**[Test build #127942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127942/testReport)** for PR 29535 at commit [`05b55cb`](https://github.com/apache/spark/commit/05b55cbb5107597fc85687c355ecb0b983605f8e).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681700064
LGTM except 2 minor comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682062496
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682059385
**[Test build #127961 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127961/testReport)** for PR 29535 at commit [`a50bf2f`](https://github.com/apache/spark/commit/a50bf2f83afca1929c989734c68b795c225c3a65).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-679427286
**[Test build #127860 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127860/testReport)** for PR 29535 at commit [`b54cc55`](https://github.com/apache/spark/commit/b54cc55c85055ced0582a0e0c513c2e2c1fee504).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682043545
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682043545
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-681183024
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-682038993
**[Test build #127959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127959/testReport)** for PR 29535 at commit [`a50bf2f`](https://github.com/apache/spark/commit/a50bf2f83afca1929c989734c68b795c225c3a65).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-679427549
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29535:
URL: https://github.com/apache/spark/pull/29535#discussion_r479424087
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -544,6 +545,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
case None => Nil
case Some(null) => Nil
case Some(any) => any :: Nil
+ case map: CaseInsensitiveStringMap =>
+ truncatedString(map.entrySet().toArray(), "[", ", ", "]", maxFields) :: Nil
Review comment:
Can we add a test case in `ExplainSuite`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29535:
URL: https://github.com/apache/spark/pull/29535#issuecomment-683595474
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29535:
URL: https://github.com/apache/spark/pull/29535
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org