You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2018/08/30 08:49:26 UTC
[GitHub] spark pull request #22281: [SPARK-25280][SQL] Add support for SQL syntax for...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/22281
[SPARK-25280][SQL] Add support for SQL syntax for DataSourceV2
## What changes were proposed in this pull request?
This PR targets for DataSource V2 to have `USING` syntax support mainly.
Currently,
```scala
spark.sql(s"CREATE TABLE tableB USING ${classOf[SimpleDataSourceV2].getCanonicalName}")
```
produces an error:
```
org.apache.spark.sql.sources.v2.SimpleDataSourceV2 is not a valid Spark SQL Data Source.;
org.apache.spark.sql.AnalysisException: org.apache.spark.sql.sources.v2.SimpleDataSourceV2 is not a valid Spark SQL Data Source.;
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:385)
at org.apache.spark.sql.execution.command.CreateDataSourceTableCommand.run(createDataSourceTables.scala:78)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
```
So that developers (Datasource V1) can easily migrate and their users can smoothly change their codes using `USING` syntax from Datasource v1, we better support this case as well.
There's a discussion thread about this here as well - http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-USING-syntax-for-Datasource-V2-td24754.html
Some discussions are going on:
http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-USING-syntax-for-Datasource-V2-td24754.html
For this one, looks we can proceed orthogonally with multiple catalog support.
The approach taken here is basically introduce `DataSourceRelation` trait which connects DataSourceV1 and DataSourceV2 so that the changes can be minimised. For extendability, this uses a pattern match so that, for instance, newer DataSource can be added in the (far) future.
For `StreamingDataSourceV2Relation` and `BatchWriteSupportProvider`, it is not handled here.
## How was this patch tested?
Unit tests were added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark using-syntax-dsv2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22281.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22281
----
commit 35f1748cb414990e03f0fd5802d26151a371ee13
Author: hyukjinkwon <gu...@...>
Date: 2018-08-28T07:01:16Z
Add support for SQL syntax for DataSourceV2
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22281
The catalog integration stuff looked to me able to proceed orthogonally since this change target to use existing catalog. It looked indeed kind of tricky but I thought the current way kind of resolves it rather cleanly. @gengliangwang, mind if I ask your opinion on this approach?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for SQL syntax for DataSo...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22281
cc @cloud-fan and @rdblue, WDYT about this approach?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22281
@rdblue, Thanks, I will take a look and see if we can keep the current approach for the write side too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95533/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22281
@HyukjinKwon When I implemented the migration, I met the problem you specify in the PR description.
At that time I used a fallback framework to switch the data source to V1 as temporary fix(https://github.com/apache/spark/pull/21123), since we didn't have a design ready for catalog yet.
Overall I think we should reach an agreement on data source V2 catalog first. We have been through too many changes in related code. Each time a lot of code need to be touched to implement a new proposal...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22281
**[Test build #95539 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95539/testReport)** for PR 22281 at commit [`1982899`](https://github.com/apache/spark/commit/1982899631bbd7a9ea2994bdb3f94b0c2411129c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22281
Yea, but the default fallback should rather be DataSource V2's. Both of you are super active in DataSource V2. Do you guys have some concerns about defaulting to DataSource V1's behaviour?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95539/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22281
> We have been through too many changes in related code. Each time a lot of code need to be touched to implement a new proposal...
I agree in general. Thing is, I doubt if multiple catalog support can directly relate to follow Datasource V1's behaviour. Took a look for writing path and still it's possible to have the same interface.
@rdblue, now I am getting what you meant by removing save modes - however, to me looks possible to have the current interface. Current change looks rather how to deal with provider matching within SQL to target match Scala side API to SQL APIs. Dealing with commands, new plans and save mode sounds lower level to me.
The actual codes proposed here is to have one trait to link both data sources to match Scala side to SQL side. I assume most of problems in the current DataSource V2 are already exposed in Scala side as well, no?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22281
**[Test build #95460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95460/testReport)** for PR 22281 at commit [`654ee2d`](https://github.com/apache/spark/commit/654ee2d47068d470abadd87865a285fbf00c7bef).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22281
**[Test build #95460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95460/testReport)** for PR 22281 at commit [`654ee2d`](https://github.com/apache/spark/commit/654ee2d47068d470abadd87865a285fbf00c7bef).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22281
Would you guys mind if I ask to point out some concerns that I might better have to double check by myself?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22281
We need to think of the big picture. I think one possible design is to not support USING in data source v2, and always ask data sources to provide a catalog. AFAIK there is no consensus about how end-users should use data source v2 with tables yet. We need to get a consensus first.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2737/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22281: [SPARK-25280][SQL] Add support for SQL syntax for...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22281#discussion_r213951028
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2Suite.scala ---
@@ -147,37 +172,34 @@ class DataSourceV2Suite extends QueryTest with SharedSQLContext {
}
}
- test("partitioning reporting") {
+ Seq(classOf[PartitionAwareDataSource], classOf[JavaPartitionAwareDataSource]).foreach { cls =>
import org.apache.spark.sql.functions.{count, sum}
- Seq(classOf[PartitionAwareDataSource], classOf[JavaPartitionAwareDataSource]).foreach { cls =>
- withClue(cls.getName) {
--- End diff --
The diff here is mainly indentation. I tried to reduce the diff but failed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22281
**[Test build #95459 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95459/testReport)** for PR 22281 at commit [`35f1748`](https://github.com/apache/spark/commit/35f1748cb414990e03f0fd5802d26151a371ee13).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait DataSourceRelation `
* `abstract class BaseRelation extends DataSourceRelation `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/22281
Thanks for working on this, @HyukjinKwon. I think it's great that this is getting the conversation started. I agree with @cloud-fan that we should think through how we want v2 to work.
The read path is fairly straight-forward, we just need to make sure the `ResolveRelations` analyzer rule returns a v2 relation. There are some details to talk through here as well when we think about [multi-catalog support](https://github.com/apache/spark/pull/21306), but the difficulty is mostly on the write side.
For SQL operations that result in writes, we want to use the new v2 logical plans. First, we need to add those and define exactly what those plans do so that we have documented and reliable behavior; I have a few open PRs for this. Then, I think we need a set of rules to convert from the plans produced by the parser (`CreateTable` with a `query`) to the new logical plans (`CreateTableAsSelect`) if the relation resolves to a v2 relation.
For v2 table creation (`CreateTable` without a `query`), I think the current USING syntax would work as it does today. I'd like to move this to the new catalog API when we add it, but I think that is mostly orthogonal.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22281
For clarification, I am okay with targeting this to 3.0.0 since the code freeze will be very soon if I am not mistaken.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95460/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22281
USING syntax has to be there, but what can USING maybe only data source v1 and file format.
IIUC the agreement is: a data source v2 with catalog can create a table with USING, and the data source should interpret the USING parameter. e.g. `USING parquet` may have a different meaning in iceberg data source.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22281
**[Test build #95533 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95533/testReport)** for PR 22281 at commit [`5a49f0d`](https://github.com/apache/spark/commit/5a49f0d50ca2f7a0446e91409c232dcb33ca3bae).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait DataSourceRelation `
* `abstract class BaseRelation extends DataSourceRelation `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22281
how is this related to the catalog integration work @rdblue is working on? IIRC @gengliangwang skipped this part when migrating file source because there are some problems hard to solve.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for SQL syntax for DataSo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2695/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22281
**[Test build #95539 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95539/testReport)** for PR 22281 at commit [`1982899`](https://github.com/apache/spark/commit/1982899631bbd7a9ea2994bdb3f94b0c2411129c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95459/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2696/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22281
**[Test build #95533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95533/testReport)** for PR 22281 at commit [`5a49f0d`](https://github.com/apache/spark/commit/5a49f0d50ca2f7a0446e91409c232dcb33ca3bae).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for SQL syntax for DataSo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for SQL syntax for DataSo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22281
**[Test build #95459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95459/testReport)** for PR 22281 at commit [`35f1748`](https://github.com/apache/spark/commit/35f1748cb414990e03f0fd5802d26151a371ee13).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2731/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22281
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22281
> I think one possible design is to not support USING in data source v2, and always ask data sources to provide a catalog.
I thought we almost all agreed with supporting USING syntax. if we shouldn't, then we shouldn't have a `FileFormat` compatibility too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org