You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/12 14:02:45 UTC
[GitHub] [spark] cloud-fan opened a new pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
cloud-fan opened a new pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
`INSERT OVERWRITE DIRECTORY` can only use file format (class implements `org.apache.spark.sql.execution.datasources.FileFormat`). This PR fixes it and other minor improvement.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598204078
cc @dongjoon-hyun @maropu @viirya
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598207731
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24446/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27891: [SPARK-31133][SQL][DOC]
fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599430545
**[Test build #119857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119857/testReport)** for PR 27891 at commit [`f58c2d3`](https://github.com/apache/spark/commit/f58c2d3f29e1f0325e2493dedd78d3719eceee8e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599431191
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24587/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599430545
**[Test build #119857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119857/testReport)** for PR 27891 at commit [`f58c2d3`](https://github.com/apache/spark/commit/f58c2d3f29e1f0325e2493dedd78d3719eceee8e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391643394
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -92,7 +92,7 @@ LOAD DATA [ LOCAL ] INPATH path [ OVERWRITE ] INTO TABLE table_identifier [ part
+ -------------- + ------------------------------ + -------------- +
-- Example with partition specification.
- CREATE TABLE test_partition (c1 INT, c2 INT, c3 INT) USING HIVE PARTITIONED BY (c2, c3);
+ CREATE TABLE test_partition (c1 INT, c2 INT, c3 INT) PARTITIONED BY (c2, c3);
Review comment:
This is used as the source so doesn't need to be a hive table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391643102
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -78,7 +78,7 @@ LOAD DATA [ LOCAL ] INPATH path [ OVERWRITE ] INTO TABLE table_identifier [ part
| Amy Smith | 123 Park Ave, San Jose | 111111 |
+ -------------- + ------------------------------ + -------------- +
- CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT);
+ CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT) USING HIVE;
Review comment:
This must be a hive table
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599431175
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27891: [SPARK-31133][SQL][DOC]
fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598206400
**[Test build #119717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119717/testReport)** for PR 27891 at commit [`4be92bb`](https://github.com/apache/spark/commit/4be92bbe1371b5a316fdbb66dc2e2f2562a3daf5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r392017686
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -78,7 +78,7 @@ LOAD DATA [ LOCAL ] INPATH path [ OVERWRITE ] INTO TABLE table_identifier [ part
| Amy Smith | 123 Park Ave, San Jose | 111111 |
+ -------------- + ------------------------------ + -------------- +
- CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT);
+ CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT) USING HIVE;
Review comment:
> The benefit is just saving a few word USING PARQUET or something.
This is a completely underestimate. Please see my comment in https://github.com/apache/spark/pull/27894#issuecomment-598534428
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r392020683
##########
File path: docs/sql-ref-syntax-dml-insert-overwrite-directory.md
##########
@@ -39,13 +39,13 @@ INSERT OVERWRITE [ LOCAL ] DIRECTORY [ directory_path ]
<dl>
<dt><code><em>file_format</em></code></dt>
<dd>
- Specifies the file format to use for the insert. Valid options are <code>TEXT</code>, <code>CSV</code>, <code>JSON</code>, <code>JDBC</code>, <code>PARQUET</code>, <code>ORC</code>, <code>HIVE</code>, <code>DELTA</code>, <code>LIBSVM</code>, or a fully qualified class name of a custom implementation of <code>org.apache.spark.sql.sources.DataSourceRegister</code>.
+ Specifies the file format to use for the insert. Valid options are <code>TEXT</code>, <code>CSV</code>, <code>JSON</code>, <code>JDBC</code>, <code>PARQUET</code>, <code>ORC</code>, <code>HIVE</code>, <code>DELTA</code>, <code>LIBSVM</code>, or a fully qualified class name of a custom implementation of <code>org.apache.spark.sql.execution.datasources.FileFormat</code>.
Review comment:
Isn't AVRO also valid? Btw, does DELTA implement FileFormat? If it means DeltaDataSource, looks like it only implements DataSourceRegister.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-600630204
I'll merge it in a few days if no objection. The CHAR/VARCHAR discussion is still going on in dev list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598207716
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598207731
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24446/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598215138
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119717/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599431191
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24587/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391813292
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -78,7 +78,7 @@ LOAD DATA [ LOCAL ] INPATH path [ OVERWRITE ] INTO TABLE table_identifier [ part
| Amy Smith | 123 Park Ave, San Jose | 111111 |
+ -------------- + ------------------------------ + -------------- +
- CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT);
+ CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT) USING HIVE;
Review comment:
Please don't get me wrong. You know that I loved that patch and [tried to minimize the impact while embracing it](https://lists.apache.org/thread.html/493f88c10169680191791f9f6962fd16cd0ffa3b06726e92ed04cbe1%40%3Cdev.spark.apache.org%3E). However, the new policy is designed to ban this kind of behavior change (SPARK-30098). Technically,
1. The benefit is just saving a few word `USING PARQUET` or something.
2. The downside is breaking the existing user pipelines.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391754858
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -78,7 +78,7 @@ LOAD DATA [ LOCAL ] INPATH path [ OVERWRITE ] INTO TABLE table_identifier [ part
| Amy Smith | 123 Park Ave, San Jose | 111111 |
+ -------------- + ------------------------------ + -------------- +
- CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT);
+ CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT) USING HIVE;
Review comment:
Hi, @cloud-fan . This reminds me.
```
SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
```
@marmbrus and @gatorsmile . Do we need to revert SPARK-30098 due to its silent behavior change?
Also, cc @rxin since he is the release manager for 3.0.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599431175
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r392877044
##########
File path: docs/sql-ref-syntax-dml-insert-overwrite-directory.md
##########
@@ -39,13 +39,13 @@ INSERT OVERWRITE [ LOCAL ] DIRECTORY [ directory_path ]
<dl>
<dt><code><em>file_format</em></code></dt>
<dd>
- Specifies the file format to use for the insert. Valid options are <code>TEXT</code>, <code>CSV</code>, <code>JSON</code>, <code>JDBC</code>, <code>PARQUET</code>, <code>ORC</code>, <code>HIVE</code>, <code>DELTA</code>, <code>LIBSVM</code>, or a fully qualified class name of a custom implementation of <code>org.apache.spark.sql.sources.DataSourceRegister</code>.
+ Specifies the file format to use for the insert. Valid options are <code>TEXT</code>, <code>CSV</code>, <code>JSON</code>, <code>JDBC</code>, <code>PARQUET</code>, <code>ORC</code>, <code>HIVE</code>, <code>DELTA</code>, <code>LIBSVM</code>, or a fully qualified class name of a custom implementation of <code>org.apache.spark.sql.execution.datasources.FileFormat</code>.
Review comment:
Avro implements `FileFormat`. I don't know if DELTA is supported in `INSERT OVERWRITE [ LOCAL ] DIRECTORY`, let me remove it as it's an external source.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598215138
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119717/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598215127
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391642129
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -20,7 +20,7 @@ license: |
---
### Description
-`LOAD DATA` statement loads the data into a table from the user specified directory or file. If a directory is specified then all the files from the directory are loaded. If a file is specified then only the single file is loaded. Additionally the `LOAD DATA` statement takes an optional partition specification. When a partition is specified, the data files (when input source is a directory) or the single file (when input source is a file) are loaded into the partition of the target table.
+`LOAD DATA` statement loads the data into a Hive serde table from the user specified directory or file. If a directory is specified then all the files from the directory are loaded. If a file is specified then only the single file is loaded. Additionally the `LOAD DATA` statement takes an optional partition specification. When a partition is specified, the data files (when input source is a directory) or the single file (when input source is a file) are loaded into the partition of the target table.
Review comment:
``LOAD DATA` only works for hive table
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391760256
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -78,7 +78,7 @@ LOAD DATA [ LOCAL ] INPATH path [ OVERWRITE ] INTO TABLE table_identifier [ part
| Amy Smith | 123 Park Ave, San Jose | 111111 |
+ -------------- + ------------------------------ + -------------- +
- CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT);
+ CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT) USING HIVE;
Review comment:
I filed SPARK-31136 to track the discussion and the final result.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391754858
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -78,7 +78,7 @@ LOAD DATA [ LOCAL ] INPATH path [ OVERWRITE ] INTO TABLE table_identifier [ part
| Amy Smith | 123 Park Ave, San Jose | 111111 |
+ -------------- + ------------------------------ + -------------- +
- CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT);
+ CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT) USING HIVE;
Review comment:
Hi, @cloud-fan . This reminds me the following.
```
SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
```
@marmbrus and @gatorsmile . Do we need to revert SPARK-30098 due to its silent behavior change?
Also, cc @rxin since he is the release manager for 3.0.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598207716
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391813292
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -78,7 +78,7 @@ LOAD DATA [ LOCAL ] INPATH path [ OVERWRITE ] INTO TABLE table_identifier [ part
| Amy Smith | 123 Park Ave, San Jose | 111111 |
+ -------------- + ------------------------------ + -------------- +
- CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT);
+ CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT) USING HIVE;
Review comment:
Please don't get me wrong. You know that I loved that patch. However, the new policy is designed to ban this kind of behavior change (SPARK-30098). Technically,
1. The benefit is just saving a few word `USING PARQUET` or something.
2. The downside is breaking the existing user pipelines.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-602609682
merging to master/3.0, thanks for review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599437018
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119857/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391768444
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -78,7 +78,7 @@ LOAD DATA [ LOCAL ] INPATH path [ OVERWRITE ] INTO TABLE table_identifier [ part
| Amy Smith | 123 Park Ave, San Jose | 111111 |
+ -------------- + ------------------------------ + -------------- +
- CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT);
+ CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT) USING HIVE;
Review comment:
The cost of break is mostly users can't run `LOAD TABLE`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27891: [SPARK-31133][SQL][DOC]
fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598214886
**[Test build #119717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119717/testReport)** for PR 27891 at commit [`4be92bb`](https://github.com/apache/spark/commit/4be92bbe1371b5a316fdbb66dc2e2f2562a3daf5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598215127
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599437006
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27891: [SPARK-31133][SQL][DOC]
fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599436820
**[Test build #119857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119857/testReport)** for PR 27891 at commit [`f58c2d3`](https://github.com/apache/spark/commit/f58c2d3f29e1f0325e2493dedd78d3719eceee8e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599437006
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599430099
shall we merge this PR first? It correctly describes the current behavior. If we want to change the behavior, we should update the document accordingly, instead of blocking this PR and wait for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391839894
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -78,7 +78,7 @@ LOAD DATA [ LOCAL ] INPATH path [ OVERWRITE ] INTO TABLE table_identifier [ part
| Amy Smith | 123 Park Ave, San Jose | 111111 |
+ -------------- + ------------------------------ + -------------- +
- CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT);
+ CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT) USING HIVE;
Review comment:
I updated the JIRA description (SPARK-31136) with @cloud-fan 's example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598206400
**[Test build #119717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119717/testReport)** for PR 27891 at commit [`4be92bb`](https://github.com/apache/spark/commit/4be92bbe1371b5a316fdbb66dc2e2f2562a3daf5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r392020972
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -20,7 +20,7 @@ license: |
---
### Description
-`LOAD DATA` statement loads the data into a table from the user specified directory or file. If a directory is specified then all the files from the directory are loaded. If a file is specified then only the single file is loaded. Additionally the `LOAD DATA` statement takes an optional partition specification. When a partition is specified, the data files (when input source is a directory) or the single file (when input source is a file) are loaded into the partition of the target table.
+`LOAD DATA` statement loads the data into a Hive serde table from the user specified directory or file. If a directory is specified then all the files from the directory are loaded. If a file is specified then only the single file is loaded. Additionally the `LOAD DATA` statement takes an optional partition specification. When a partition is specified, the data files (when input source is a directory) or the single file (when input source is a file) are loaded into the partition of the target table.
Review comment:
good catch!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391839894
##########
File path: docs/sql-ref-syntax-dml-load.md
##########
@@ -78,7 +78,7 @@ LOAD DATA [ LOCAL ] INPATH path [ OVERWRITE ] INTO TABLE table_identifier [ part
| Amy Smith | 123 Park Ave, San Jose | 111111 |
+ -------------- + ------------------------------ + -------------- +
- CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT);
+ CREATE TABLE test_load (name VARCHAR(64), address VARCHAR(64), student_id INT) USING HIVE;
Review comment:
I updated the JIRA description with @cloud-fan 's example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598540810
Hi, All. I marked SPARK-31136 as a correctness issue and added the example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-599437018
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119857/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#discussion_r391668117
##########
File path: docs/sql-ref-syntax-dml-insert-overwrite-directory.md
##########
@@ -9,17 +9,17 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
-
+
http://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
### Description
-The `INSERT OVERWRITE DIRECTORY` statement overwrites the existing data in the directory with the new values using Spark native format. The inserted rows can be specified by value expressions or result from a query.
+The `INSERT OVERWRITE DIRECTORY` statement overwrites the existing data in the directory with the new values using Spark file format. The inserted rows can be specified by value expressions or result from a query.
Review comment:
nit: `Spark file format` -> `a given Spark file format`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27891:
[SPARK-31133][SQL][DOC] fix sql ref doc for DML
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27891: [SPARK-31133][SQL][DOC] fix sql ref doc for DML
URL: https://github.com/apache/spark/pull/27891#issuecomment-598540810
Hi, All. I marked SPARK-31136 as a correctness issue and added the example of `CHAR` type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org