You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/05/25 08:08:35 UTC
[GitHub] [spark] itholic opened a new pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
itholic opened a new pull request #32660:
URL: https://github.com/apache/spark/pull/32660
### What changes were proposed in this pull request?
This PR proposes move text data source options from Python, Scala and Java into a single page.
### Why are the changes needed?
So far, the documentation for text data source options is separated into different pages for each language API documents. However, this makes managing many options inconvenient, so it is efficient to manage all options in a single page and provide a link to that page in the API of each language.
### Does this PR introduce _any_ user-facing change?
Yes, the documents will be shown below after this change:
- "Text Files" page
- Python
<img width="791" alt="Screen Shot 2021-05-25 at 5 04 26 PM" src="https://user-images.githubusercontent.com/44108233/119462469-b9c11d00-bd7b-11eb-8f19-2ba7b9ceb318.png">
- Scala
<img width="683" alt="Screen Shot 2021-05-25 at 5 05 10 PM" src="https://user-images.githubusercontent.com/44108233/119462483-bd54a400-bd7b-11eb-8177-74e4d7035e63.png">
- Java
<img width="665" alt="Screen Shot 2021-05-25 at 5 05 36 PM" src="https://user-images.githubusercontent.com/44108233/119462501-bfb6fe00-bd7b-11eb-8161-12c58fabe7e2.png">
### How was this patch tested?
Manually build docs and confirm the page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848677049
**[Test build #138965 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138965/testReport)** for PR 32660 at commit [`f03c03d`](https://github.com/apache/spark/commit/f03c03dfc9ede54920c642f9129560f871c38303).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r638680715
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of CSV can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>wholetext</code></td>
+ <td>None</td>
+ <td>If true, read each file from input path(s) as a single row.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>lineSep</code></td>
+ <td>None</td>
+ <td>Defines the line separator that should be used for parsing. If None is set, it covers all <code>\\r</code>, <code>\\r\\n</code> and <code>\\n</code>. Maximum length is 1 character.</td>
Review comment:
Looks like we lost the description for writing "defines the line separator that should be used for writing"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848528274
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43484/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848444125
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43472/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-847749695
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43436/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848423791
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43472/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848548880
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43484/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-847868397
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138915/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848389273
**[Test build #138949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138949/testReport)** for PR 32660 at commit [`1ee72e2`](https://github.com/apache/spark/commit/1ee72e207b56a5c64d4b6f84f8531f13391f02ac).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639392790
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of CSV can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>wholetext</code></td>
+ <td>None</td>
Review comment:
@HyukjinKwon Sorry, SPARK-35523. I fixed the link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848692023
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138965/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639339008
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of text can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>wholetext</code></td>
+ <td>false</td>
+ <td>If true, read each file from input path(s) as a single row.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>lineSep</code></td>
+ <td><code>\r</code>, <code>\r\n</code>, <code>\n</code> for reading, <code>\\n</code> for writing</td>
+ <td>Defines the line separator that should be used for reading or writing.</td>
+ <td>read/write</td>
+ </tr>
+ <tr>
+ <td><code>compression</code></td>
+ <td>null</td>
Review comment:
you can use "(none)", see also https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32660:
URL: https://github.com/apache/spark/pull/32660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639549599
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,36 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of text can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+ * `OPTIONS` clause at [CREATE TABLE USING DATA_SOURCE](sql-ref-syntax-ddl-create-table-datasource.html)
Review comment:
Sorry I missed it here. This is not in "the `.option`/`.options` methods of". Can you make the level down?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848408770
**[Test build #138952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138952/testReport)** for PR 32660 at commit [`98840f3`](https://github.com/apache/spark/commit/98840f3522888db81e60d85b29c4957a71a848ff).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639339292
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of text can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>wholetext</code></td>
+ <td>false</td>
+ <td>If true, read each file from input path(s) as a single row.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>lineSep</code></td>
+ <td><code>\r</code>, <code>\r\n</code>, <code>\n</code> for reading, <code>\\n</code> for writing</td>
Review comment:
is it `\n` or `\\n`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639338072
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of CSV can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>wholetext</code></td>
+ <td>None</td>
Review comment:
SPARK-25523?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639322930
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of CSV can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>wholetext</code></td>
+ <td>None</td>
Review comment:
Okay, I just created sub-task, SPARK-25523.
I'll handle the other pages over there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-847749695
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43436/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848500657
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138949/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639430157
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of text can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
Review comment:
Just created JIRA sub-task here: SPARK-35528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639400650
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of text can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
Review comment:
Oh, got it. I'll create the new JIRA and address the other pages, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r638677411
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of CSV can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>wholetext</code></td>
+ <td>None</td>
Review comment:
Can you fix the default value properly? You can refer the docs in Scaladoc.
Looks like JSON and other sources have the same issue. Can you fix them in another Pr too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848502638
**[Test build #138965 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138965/testReport)** for PR 32660 at commit [`f03c03d`](https://github.com/apache/spark/commit/f03c03dfc9ede54920c642f9129560f871c38303).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848548911
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43484/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848506497
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138952/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848389273
**[Test build #138949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138949/testReport)** for PR 32660 at commit [`1ee72e2`](https://github.com/apache/spark/commit/1ee72e207b56a5c64d4b6f84f8531f13391f02ac).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848479476
**[Test build #138949 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138949/testReport)** for PR 32660 at commit [`1ee72e2`](https://github.com/apache/spark/commit/1ee72e207b56a5c64d4b6f84f8531f13391f02ac).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848564451
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639322930
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of CSV can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>wholetext</code></td>
+ <td>None</td>
Review comment:
Okay, I just created sub-task, SPARK-35523.
I'll handle the other pages over there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639430157
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of text can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
Review comment:
Just created JIRA here: SPARK-35528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639356857
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of text can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>wholetext</code></td>
+ <td>false</td>
+ <td>If true, read each file from input path(s) as a single row.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>lineSep</code></td>
+ <td><code>\r</code>, <code>\r\n</code>, <code>\n</code> for reading, <code>\\n</code> for writing</td>
+ <td>Defines the line separator that should be used for reading or writing.</td>
+ <td>read/write</td>
+ </tr>
+ <tr>
+ <td><code>compression</code></td>
+ <td>null</td>
Review comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848426813
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43469/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848506497
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138952/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-847856567
**[Test build #138915 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138915/testReport)** for PR 32660 at commit [`cddc556`](https://github.com/apache/spark/commit/cddc556f136400c44cdeb0a10e591259b3e8a307).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639356536
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of text can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>wholetext</code></td>
+ <td>false</td>
+ <td>If true, read each file from input path(s) as a single row.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>lineSep</code></td>
+ <td><code>\r</code>, <code>\r\n</code>, <code>\n</code> for reading, <code>\\n</code> for writing</td>
Review comment:
I'd consolidate it to `\n`, as following the Scaladoc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-847734570
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43436/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848548911
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848505156
**[Test build #138952 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138952/testReport)** for PR 32660 at commit [`98840f3`](https://github.com/apache/spark/commit/98840f3522888db81e60d85b29c4957a71a848ff).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r639396443
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of text can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
Review comment:
Can you also add CREATE TABLE with OPTIONS clause? (see https://github.com/apache/spark/pull/32546#discussion_r638688399). We will fix other pages too in another Pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848444125
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43472/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848407702
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43469/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848426813
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43469/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-847666240
**[Test build #138915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138915/testReport)** for PR 32660 at commit [`cddc556`](https://github.com/apache/spark/commit/cddc556f136400c44cdeb0a10e591259b3e8a307).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r638677761
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,35 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of CSV can be set via:
Review comment:
CSV?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848432919
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43472/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-847666240
**[Test build #138915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138915/testReport)** for PR 32660 at commit [`cddc556`](https://github.com/apache/spark/commit/cddc556f136400c44cdeb0a10e591259b3e8a307).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848418223
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43469/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848408770
**[Test build #138952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138952/testReport)** for PR 32660 at commit [`98840f3`](https://github.com/apache/spark/commit/98840f3522888db81e60d85b29c4957a71a848ff).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848502638
**[Test build #138965 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138965/testReport)** for PR 32660 at commit [`f03c03d`](https://github.com/apache/spark/commit/f03c03dfc9ede54920c642f9129560f871c38303).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-847868397
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138915/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-847709981
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43436/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32660:
URL: https://github.com/apache/spark/pull/32660#discussion_r640228432
##########
File path: docs/sql-data-sources-text.md
##########
@@ -38,3 +36,36 @@ Spark SQL provides `spark.read().text("file_name")` to read a file or directory
</div>
</div>
+
+## Data Source Option
+
+Data source options of text can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+ * `OPTIONS` clause at [CREATE TABLE USING DATA_SOURCE](sql-ref-syntax-ddl-create-table-datasource.html)
Review comment:
Sure. let me fix it in https://github.com/apache/spark/pull/32658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32660: [SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32660:
URL: https://github.com/apache/spark/pull/32660#issuecomment-848500657
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138949/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org