You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/05/14 04:59:12 UTC
[GitHub] [spark] itholic opened a new pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
itholic opened a new pull request #32546:
URL: https://github.com/apache/spark/pull/32546
### What changes were proposed in this pull request?
This PR proposes move ORC data source options from Python, Scala and Java into a single page.
### Why are the changes needed?
So far, the documentation for ORC data source options is separated into different pages for each language API documents. However, this makes managing many options inconvenient, so it is efficient to manage all options in a single page and provide a link to that page in the API of each language.
### Does this PR introduce _any_ user-facing change?
The "Data Source Option" page is added to the "ORC Files" page, and linked to the each API docs.
<img width="818" alt="Screen Shot 2021-05-14 at 1 56 32 PM" src="https://user-images.githubusercontent.com/44108233/118223516-6ec22280-b4bc-11eb-8281-847546452863.png">
### How was this patch tested?
Manually build docs and confirm the page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842385319
**[Test build #138635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138635/testReport)** for PR 32546 at commit [`aa54b45`](https://github.com/apache/spark/commit/aa54b4542e2fc4c0a5cff9db4ef7da8e7187b3d5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-844879518
**[Test build #138738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138738/testReport)** for PR 32546 at commit [`0b0e183`](https://github.com/apache/spark/commit/0b0e183d9e38bcd3f15fef8724ae69ee224f0905).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r635973164
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
* format("orc").save(path)
* }}}
*
- * You can set the following ORC-specific option(s) for writing ORC files:
- * <ul>
- * <li>`compression` (default is the value specified in `spark.sql.orc.compression.codec`):
- * compression codec to use when saving to file. This can be one of the known case-insensitive
- * shorten names(`none`, `snappy`, `zlib`, `lzo`, and `zstd`). This will override
- * `orc.compress` and `spark.sql.orc.compression.codec`. If `orc.compress` is given,
- * it overrides `spark.sql.orc.compression.codec`.</li>
- * </ul>
+ * ORC-specific option(s) for writing ORC files can be found in
+ * <a href=
+ * "https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option">
+ * Data Source Option</a> in the version you use.
Review comment:
I would just add it for simplicity, and in case we have write generic options in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845634703
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43300/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic edited a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic edited a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845706476
Thanks, @HyukjinKwon .
PR description is updated, and also the PR description of https://github.com/apache/spark/pull/32204, https://github.com/apache/spark/pull/32161 are updated as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845028495
**[Test build #138750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138750/testReport)** for PR 32546 at commit [`3971dd8`](https://github.com/apache/spark/commit/3971dd8a95a5d97b25ec8c4224cd7868ec9abe2a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845630052
Thank you for pinging me, @HyukjinKwon .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-844889966
**[Test build #138742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138742/testReport)** for PR 32546 at commit [`6358d59`](https://github.com/apache/spark/commit/6358d5940545021f96f1d9eb52ca3c674d7b01bd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845286974
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138750/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842296285
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138632/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r636615565
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -874,23 +874,10 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
/**
* Loads ORC files and returns the result as a `DataFrame`.
*
- * You can set the following ORC-specific option(s) for reading ORC files:
- * <ul>
- * <li>`mergeSchema` (default is the value specified in `spark.sql.orc.mergeSchema`): sets whether
- * we should merge schemas collected from all ORC part-files. This will override
- * `spark.sql.orc.mergeSchema`.</li>
- * <li>`pathGlobFilter`: an optional glob pattern to only include files with paths matching
- * the pattern. The syntax follows <code>org.apache.hadoop.fs.GlobFilter</code>.
- * It does not change the behavior of partition discovery.</li>
- * <li>`modifiedBefore` (batch only): an optional timestamp to only include files with
- * modification times occurring before the specified Time. The provided timestamp
- * must be in the following form: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)</li>
- * <li>`modifiedAfter` (batch only): an optional timestamp to only include files with
- * modification times occurring after the specified Time. The provided timestamp
- * must be in the following form: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)</li>
- * <li>`recursiveFileLookup`: recursively scan a directory for files. Using this option
- * disables partition discovery</li>
- * </ul>
+ * ORC-specific option(s) for reading ORC files can be found in
+ * <a href=
+ * "https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option">
Review comment:
Ditto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842781690
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43175/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845632925
Thank you, @itholic and @HyukjinKwon . The refactoring idea looks good to me. I commented only a technical issue about the link usage. I'll leave this to @HyukjinKwon .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-841037708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-844879518
**[Test build #138738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138738/testReport)** for PR 32546 at commit [`0b0e183`](https://github.com/apache/spark/commit/0b0e183d9e38bcd3f15fef8724ae69ee224f0905).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845617446
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43300/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845596172
**[Test build #138776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138776/testReport)** for PR 32546 at commit [`043d308`](https://github.com/apache/spark/commit/043d3087c838c2f6439d798bdc8e889d6a728ff7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-844978650
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43264/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845028495
**[Test build #138750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138750/testReport)** for PR 32546 at commit [`3971dd8`](https://github.com/apache/spark/commit/3971dd8a95a5d97b25ec8c4224cd7868ec9abe2a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-843185107
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43197/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842781690
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43175/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r635796796
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -874,23 +874,14 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
/**
* Loads ORC files and returns the result as a `DataFrame`.
*
- * You can set the following ORC-specific option(s) for reading ORC files:
- * <ul>
- * <li>`mergeSchema` (default is the value specified in `spark.sql.orc.mergeSchema`): sets whether
- * we should merge schemas collected from all ORC part-files. This will override
- * `spark.sql.orc.mergeSchema`.</li>
- * <li>`pathGlobFilter`: an optional glob pattern to only include files with paths matching
- * the pattern. The syntax follows <code>org.apache.hadoop.fs.GlobFilter</code>.
- * It does not change the behavior of partition discovery.</li>
- * <li>`modifiedBefore` (batch only): an optional timestamp to only include files with
- * modification times occurring before the specified Time. The provided timestamp
- * must be in the following form: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)</li>
- * <li>`modifiedAfter` (batch only): an optional timestamp to only include files with
- * modification times occurring after the specified Time. The provided timestamp
- * must be in the following form: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)</li>
- * <li>`recursiveFileLookup`: recursively scan a directory for files. Using this option
- * disables partition discovery</li>
- * </ul>
+ * ORC-specific option(s) for reading ORC files can be found in
Review comment:
ditto. it says ORC-specific options but also mentions about Generic Files Source Options later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845803030
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r636615149
##########
File path: python/pyspark/sql/readwriter.py
##########
@@ -793,28 +793,13 @@ def orc(self, path, mergeSchema=None, pathGlobFilter=None, recursiveFileLookup=N
Parameters
----------
path : str or list
- mergeSchema : str or bool, optional
- sets whether we should merge schemas collected from all
- ORC part-files. This will override ``spark.sql.orc.mergeSchema``.
- The default value is specified in ``spark.sql.orc.mergeSchema``.
- pathGlobFilter : str or bool
- an optional glob pattern to only include files with paths matching
- the pattern. The syntax follows `org.apache.hadoop.fs.GlobFilter`.
- It does not change the behavior of
- `partition discovery <https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#partition-discovery>`_. # noqa
- recursiveFileLookup : str or bool
- recursively scan a directory for files. Using this option
- disables
- `partition discovery <https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#partition-discovery>`_. # noqa
- modification times occurring before the specified time. The provided timestamp
- must be in the following format: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)
- modifiedBefore : an optional timestamp to only include files with
- modification times occurring before the specified time. The provided timestamp
- must be in the following format: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)
- modifiedAfter : an optional timestamp to only include files with
- modification times occurring after the specified time. The provided timestamp
- must be in the following format: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)
+ Other Parameters
+ ----------------
+ Extra options
+ For the extra options, refer to
+ `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option>`_ # noqa
Review comment:
Ditto. Can we have a more robust link here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845727564
**[Test build #138776 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138776/testReport)** for PR 32546 at commit [`043d308`](https://github.com/apache/spark/commit/043d3087c838c2f6439d798bdc8e889d6a728ff7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842859771
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138654/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842385319
**[Test build #138635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138635/testReport)** for PR 32546 at commit [`aa54b45`](https://github.com/apache/spark/commit/aa54b4542e2fc4c0a5cff9db4ef7da8e7187b3d5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842796298
Jenkins, retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r635796892
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
* format("orc").save(path)
* }}}
*
- * You can set the following ORC-specific option(s) for writing ORC files:
- * <ul>
- * <li>`compression` (default is the value specified in `spark.sql.orc.compression.codec`):
- * compression codec to use when saving to file. This can be one of the known case-insensitive
- * shorten names(`none`, `snappy`, `zlib`, `lzo`, and `zstd`). This will override
- * `orc.compress` and `spark.sql.orc.compression.codec`. If `orc.compress` is given,
- * it overrides `spark.sql.orc.compression.codec`.</li>
- * </ul>
+ * ORC-specific option(s) for writing ORC files can be found in
+ * <a href=
+ * "https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option">
+ * Data Source Option</a> in the version you use.
Review comment:
Here you didn't link generic options but you did for Parquet. What's diff?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845742460
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138776/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r635890871
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
* format("orc").save(path)
* }}}
*
- * You can set the following ORC-specific option(s) for writing ORC files:
- * <ul>
- * <li>`compression` (default is the value specified in `spark.sql.orc.compression.codec`):
- * compression codec to use when saving to file. This can be one of the known case-insensitive
- * shorten names(`none`, `snappy`, `zlib`, `lzo`, and `zstd`). This will override
- * `orc.compress` and `spark.sql.orc.compression.codec`. If `orc.compress` is given,
- * it overrides `spark.sql.orc.compression.codec`.</li>
- * </ul>
+ * ORC-specific option(s) for writing ORC files can be found in
+ * <a href=
+ * "https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option">
+ * Data Source Option</a> in the version you use.
Review comment:
@HyukjinKwon , this is because the options I removed doesn't include generic options. (It has only one option, `compression` in this case)
Would it be better to just add the link to the Generic options no matter what we removed ??
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
* format("orc").save(path)
* }}}
*
- * You can set the following ORC-specific option(s) for writing ORC files:
- * <ul>
- * <li>`compression` (default is the value specified in `spark.sql.orc.compression.codec`):
- * compression codec to use when saving to file. This can be one of the known case-insensitive
- * shorten names(`none`, `snappy`, `zlib`, `lzo`, and `zstd`). This will override
- * `orc.compress` and `spark.sql.orc.compression.codec`. If `orc.compress` is given,
- * it overrides `spark.sql.orc.compression.codec`.</li>
- * </ul>
+ * ORC-specific option(s) for writing ORC files can be found in
+ * <a href=
+ * "https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option">
+ * Data Source Option</a> in the version you use.
Review comment:
@HyukjinKwon , this is because the options I removed doesn't include generic options. (For example it has only one option, `compression` in this case)
Would it be better to just add the link to the Generic options no matter what we removed ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r636614962
##########
File path: docs/sql-data-sources-orc.md
##########
@@ -172,3 +172,29 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC
<td>2.0.0</td>
</tr>
</table>
+
+## Data Source Option
+
+Data source options of ORC can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>mergeSchema</code></td>
+ <td>None</td>
+ <td>sets whether we should merge schemas collected from all ORC part-files. This will override <code>spark.sql.orc.mergeSchema</code>. The default value is specified in <code>spark.sql.orc.mergeSchema</code>.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>compression</code></td>
+ <td>None</td>
+ <td>compression codec to use when saving to file. This can be one of the known case-insensitive shorten names (none, snappy, zlib, lzo, and zstd). This will override <code>orc.compress</code> and <code>spark.sql.orc.compression.codec</code>. If None is set, it uses the value specified in <code>spark.sql.orc.compression.codec</code>.</td>
+ <td>write</td>
+ </tr>
+</table>
+Other generic options can be found in <a href="https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html"> Generic File Source Options</a>.
Review comment:
Although I know that this is inherited, `https://spark.apache.org/docs/latest/` looks fragile to me because it is going to be a broken link when we cut `branch-3.2` on July 1st. In `branch-3.2`, it should point `3.2` document only. Shall we use a relative link instead of `/latest/`?
Like this PR, we don't know what refactoring happens in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842556737
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138635/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842979560
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138661/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845742460
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138776/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r636614962
##########
File path: docs/sql-data-sources-orc.md
##########
@@ -172,3 +172,29 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC
<td>2.0.0</td>
</tr>
</table>
+
+## Data Source Option
+
+Data source options of ORC can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>mergeSchema</code></td>
+ <td>None</td>
+ <td>sets whether we should merge schemas collected from all ORC part-files. This will override <code>spark.sql.orc.mergeSchema</code>. The default value is specified in <code>spark.sql.orc.mergeSchema</code>.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>compression</code></td>
+ <td>None</td>
+ <td>compression codec to use when saving to file. This can be one of the known case-insensitive shorten names (none, snappy, zlib, lzo, and zstd). This will override <code>orc.compress</code> and <code>spark.sql.orc.compression.codec</code>. If None is set, it uses the value specified in <code>spark.sql.orc.compression.codec</code>.</td>
+ <td>write</td>
+ </tr>
+</table>
+Other generic options can be found in <a href="https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html"> Generic File Source Options</a>.
Review comment:
`https://spark.apache.org/docs/latest/` looks fragile to me because it is going to be a broken link when we cut `branch-3.2` on July 1st. In `branch-3.2`, it should point `3.2` document only. Shall we use a relative link instead of `/latest/`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-841040361
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43067/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845165258
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43272/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r638688399
##########
File path: docs/sql-data-sources-orc.md
##########
@@ -172,3 +172,29 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC
<td>2.0.0</td>
</tr>
</table>
+
+## Data Source Option
+
+Data source options of ORC can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
Review comment:
also mention:
```
* `OPTIONS` clause at [CREATE TABLE USING DATA_SOURCE](sql-ref-syntax-ddl-create-table-datasource.html)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r638683212
##########
File path: docs/sql-data-sources-orc.md
##########
@@ -172,3 +172,29 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC
<td>2.0.0</td>
</tr>
</table>
+
+## Data Source Option
+
+Data source options of ORC can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>mergeSchema</code></td>
+ <td>None</td>
+ <td>sets whether we should merge schemas collected from all ORC part-files. This will override <code>spark.sql.orc.mergeSchema</code>. The default value is specified in <code>spark.sql.orc.mergeSchema</code>.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>compression</code></td>
+ <td>None</td>
Review comment:
In this case when the default value doesn't exist, you can follow https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration `(none)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r638688399
##########
File path: docs/sql-data-sources-orc.md
##########
@@ -172,3 +172,29 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC
<td>2.0.0</td>
</tr>
</table>
+
+## Data Source Option
+
+Data source options of ORC can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
Review comment:
also mention:
```
Also probably we should say about CREATE ... OPTIONS syntax:
```
* `OPTIONS` clause at [CREATE TABLE USING DATA_SOURCE](sql-ref-syntax-ddl-create-table-datasource.html)
```
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845104906
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138738/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842296261
**[Test build #138632 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138632/testReport)** for PR 32546 at commit [`005d87d`](https://github.com/apache/spark/commit/005d87d34dc6a67b6071852d6179040c686234d9).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-844937566
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43261/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842960974
**[Test build #138661 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138661/testReport)** for PR 32546 at commit [`2e0787c`](https://github.com/apache/spark/commit/2e0787c9763d00fd6db5794282854fa8eb94283a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-844971083
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43264/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842802735
**[Test build #138661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138661/testReport)** for PR 32546 at commit [`2e0787c`](https://github.com/apache/spark/commit/2e0787c9763d00fd6db5794282854fa8eb94283a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-844889966
**[Test build #138742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138742/testReport)** for PR 32546 at commit [`6358d59`](https://github.com/apache/spark/commit/6358d5940545021f96f1d9eb52ca3c674d7b01bd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845192779
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43272/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845144925
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138742/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845144925
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138742/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845632925
Thank you, @itholic and @HyukjinKwon . The refactoring idea looks good to me. I commented only a technical issue about the link usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r638682522
##########
File path: docs/sql-data-sources-orc.md
##########
@@ -172,3 +172,29 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC
<td>2.0.0</td>
</tr>
</table>
+
+## Data Source Option
+
+Data source options of ORC can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>mergeSchema</code></td>
+ <td>None</td>
Review comment:
@itholic it has the same issue. The default value isn't `None` but `false`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842334467
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43151/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-843101835
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43197/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842821200
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43182/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842802735
**[Test build #138661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138661/testReport)** for PR 32546 at commit [`2e0787c`](https://github.com/apache/spark/commit/2e0787c9763d00fd6db5794282854fa8eb94283a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r635974286
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala
##########
@@ -453,20 +453,21 @@ final class DataStreamReader private[sql](sparkSession: SparkSession) extends Lo
/**
* Loads a ORC file stream, returning the result as a `DataFrame`.
*
- * You can set the following ORC-specific option(s) for reading ORC files:
+ * You can set the following structured streaming option(s):
Review comment:
```suggestion
* You can set the following option(s):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845082463
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43272/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842762380
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43175/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845596172
**[Test build #138776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138776/testReport)** for PR 32546 at commit [`043d308`](https://github.com/apache/spark/commit/043d3087c838c2f6439d798bdc8e889d6a728ff7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-843064570
**[Test build #138676 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138676/testReport)** for PR 32546 at commit [`eca1295`](https://github.com/apache/spark/commit/eca1295ef2d82da331c86971b7fbf97374f10753).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842295245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845104906
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138738/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-841016889
**[Test build #138548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138548/testReport)** for PR 32546 at commit [`0ce01d0`](https://github.com/apache/spark/commit/0ce01d037a01e45b4b02ba3dbd68dfea940ecaee).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-841040361
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43067/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-841159278
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842416025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842328523
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43151/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r636569400
##########
File path: python/pyspark/sql/readwriter.py
##########
@@ -793,28 +793,15 @@ def orc(self, path, mergeSchema=None, pathGlobFilter=None, recursiveFileLookup=N
Parameters
----------
path : str or list
- mergeSchema : str or bool, optional
- sets whether we should merge schemas collected from all
- ORC part-files. This will override ``spark.sql.orc.mergeSchema``.
- The default value is specified in ``spark.sql.orc.mergeSchema``.
- pathGlobFilter : str or bool
- an optional glob pattern to only include files with paths matching
- the pattern. The syntax follows `org.apache.hadoop.fs.GlobFilter`.
- It does not change the behavior of
- `partition discovery <https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#partition-discovery>`_. # noqa
- recursiveFileLookup : str or bool
- recursively scan a directory for files. Using this option
- disables
- `partition discovery <https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#partition-discovery>`_. # noqa
- modification times occurring before the specified time. The provided timestamp
- must be in the following format: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)
- modifiedBefore : an optional timestamp to only include files with
- modification times occurring before the specified time. The provided timestamp
- must be in the following format: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)
- modifiedAfter : an optional timestamp to only include files with
- modification times occurring after the specified time. The provided timestamp
- must be in the following format: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)
+ Other Parameters
+ ----------------
+ Extra options
+ For the extra options, refer to
+ `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option>`_ # noqa
+ and
+ `Generic File Source Options <https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html`>_ # noqa
Review comment:
Shall we remove this too?
##########
File path: python/pyspark/sql/streaming.py
##########
@@ -637,20 +637,14 @@ def orc(self, path, mergeSchema=None, pathGlobFilter=None, recursiveFileLookup=N
.. versionadded:: 2.3.0
- Parameters
- ----------
- mergeSchema : str or bool, optional
- sets whether we should merge schemas collected from all
- ORC part-files. This will override ``spark.sql.orc.mergeSchema``.
- The default value is specified in ``spark.sql.orc.mergeSchema``.
- pathGlobFilter : str or bool, optional
- an optional glob pattern to only include files with paths matching
- the pattern. The syntax follows `org.apache.hadoop.fs.GlobFilter`.
- It does not change the behavior of `partition discovery`_.
- recursiveFileLookup : str or bool, optional
- recursively scan a directory for files. Using this option
- disables
- `partition discovery <https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#partition-discovery>`_. # noqa
+ Other Parameters
+ ----------------
+ Extra options
+ For the extra options, refer to
+ `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option>`_ # noqa
+ and
+ `Generic File Source Options <https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html`>_ # noqa
Review comment:
Shall we remove this too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r636614962
##########
File path: docs/sql-data-sources-orc.md
##########
@@ -172,3 +172,29 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC
<td>2.0.0</td>
</tr>
</table>
+
+## Data Source Option
+
+Data source options of ORC can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>mergeSchema</code></td>
+ <td>None</td>
+ <td>sets whether we should merge schemas collected from all ORC part-files. This will override <code>spark.sql.orc.mergeSchema</code>. The default value is specified in <code>spark.sql.orc.mergeSchema</code>.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>compression</code></td>
+ <td>None</td>
+ <td>compression codec to use when saving to file. This can be one of the known case-insensitive shorten names (none, snappy, zlib, lzo, and zstd). This will override <code>orc.compress</code> and <code>spark.sql.orc.compression.codec</code>. If None is set, it uses the value specified in <code>spark.sql.orc.compression.codec</code>.</td>
+ <td>write</td>
+ </tr>
+</table>
+Other generic options can be found in <a href="https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html"> Generic File Source Options</a>.
Review comment:
Although I know that this is inherited, `https://spark.apache.org/docs/latest/` looks fragile to me because it is going to be a broken link when we cut `branch-3.2` on July 1st. In `branch-3.2`, it should point `3.2` document only. Shall we use a relative link instead of `/latest/`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842742900
**[Test build #138654 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138654/testReport)** for PR 32546 at commit [`3740c52`](https://github.com/apache/spark/commit/3740c52865a867e7b3e6904a0fefc8304870438b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r632384490
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -909,13 +909,10 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
* }}}
* The text files will be encoded as UTF-8.
*
- * You can set the following option(s) for writing text files:
- * <ul>
- * <li>`compression` (default `null`): compression codec to use when saving to file. This can be
- * one of the known case-insensitive shorten names (`none`, `bzip2`, `gzip`, `lz4`,
- * `snappy` and `deflate`). </li>
- * <li>`lineSep` (default `\n`): defines the line separator that should be used for writing.</li>
Review comment:
this isn't `orc`. It's `text`.
##########
File path: docs/sql-data-sources-orc.md
##########
@@ -172,3 +172,32 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC
<td>2.0.0</td>
</tr>
</table>
+
+## Data Source Option
+
+Data source options of ORC can be set via:
+* the `.option`/`.options` methods of `DataFrameReader` or `DataFrameWriter`
+* the `.option`/`.options` methods of `DataStreamReader` or `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>mergeSchema</code></td>
+ <td>None</td>
+ <td>sets whether we should merge schemas collected from all ORC part-files. This will override <code>spark.sql.orc.mergeSchema</code>. The default value is specified in <code>spark.sql.orc.mergeSchema</code>.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>compression</code></td>
+ <td>None</td>
+ <td>compression codec to use when saving to file. This can be one of the known case-insensitive shorten names (none, snappy, zlib, lzo, and zstd). This will override <code>orc.compress</code> and <code>spark.sql.orc.compression.codec</code>. If None is set, it uses the value specified in <code>spark.sql.orc.compression.codec</code>.</td>
+ <td>write</td>
+ </tr>
+ <tr>
+ <td><code>lineSep</code></td>
Review comment:
Can you remove this? it's text source option, not ORC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845634703
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43300/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842295245
**[Test build #138632 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138632/testReport)** for PR 32546 at commit [`005d87d`](https://github.com/apache/spark/commit/005d87d34dc6a67b6071852d6179040c686234d9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845006185
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43261/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842334467
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43151/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r635890871
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
* format("orc").save(path)
* }}}
*
- * You can set the following ORC-specific option(s) for writing ORC files:
- * <ul>
- * <li>`compression` (default is the value specified in `spark.sql.orc.compression.codec`):
- * compression codec to use when saving to file. This can be one of the known case-insensitive
- * shorten names(`none`, `snappy`, `zlib`, `lzo`, and `zstd`). This will override
- * `orc.compress` and `spark.sql.orc.compression.codec`. If `orc.compress` is given,
- * it overrides `spark.sql.orc.compression.codec`.</li>
- * </ul>
+ * ORC-specific option(s) for writing ORC files can be found in
+ * <a href=
+ * "https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option">
+ * Data Source Option</a> in the version you use.
Review comment:
@HyukjinKwon , this is because the options I removed doesn't include generic options.
Would it be better to just add the link to the Generic options no matter what we removed ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842429041
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43155/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842859771
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138654/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842742900
**[Test build #138654 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138654/testReport)** for PR 32546 at commit [`3740c52`](https://github.com/apache/spark/commit/3740c52865a867e7b3e6904a0fefc8304870438b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845192779
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43272/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-843250684
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138676/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842556737
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138635/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842296285
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138632/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842296285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845629618
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43300/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r636625094
##########
File path: docs/sql-data-sources-orc.md
##########
@@ -172,3 +172,29 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC
<td>2.0.0</td>
</tr>
</table>
+
+## Data Source Option
+
+Data source options of ORC can be set via:
+* the `.option`/`.options` methods of
+ * `DataFrameReader`
+ * `DataFrameWriter`
+ * `DataStreamReader`
+ * `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>mergeSchema</code></td>
+ <td>None</td>
+ <td>sets whether we should merge schemas collected from all ORC part-files. This will override <code>spark.sql.orc.mergeSchema</code>. The default value is specified in <code>spark.sql.orc.mergeSchema</code>.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>compression</code></td>
+ <td>None</td>
+ <td>compression codec to use when saving to file. This can be one of the known case-insensitive shorten names (none, snappy, zlib, lzo, and zstd). This will override <code>orc.compress</code> and <code>spark.sql.orc.compression.codec</code>. If None is set, it uses the value specified in <code>spark.sql.orc.compression.codec</code>.</td>
+ <td>write</td>
+ </tr>
+</table>
+Other generic options can be found in <a href="https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html"> Generic File Source Options</a>.
Review comment:
Thanks, @dongjoon-hyun .
I took a look for that but seems tricky to create a link for each release in Scaladoc ..
I created a JIRA to track it separately here: SPARK-35481.
I will take a separate look if that's fine to you too!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-841149743
**[Test build #138548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138548/testReport)** for PR 32546 at commit [`0ce01d0`](https://github.com/apache/spark/commit/0ce01d037a01e45b4b02ba3dbd68dfea940ecaee).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r635890871
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
* format("orc").save(path)
* }}}
*
- * You can set the following ORC-specific option(s) for writing ORC files:
- * <ul>
- * <li>`compression` (default is the value specified in `spark.sql.orc.compression.codec`):
- * compression codec to use when saving to file. This can be one of the known case-insensitive
- * shorten names(`none`, `snappy`, `zlib`, `lzo`, and `zstd`). This will override
- * `orc.compress` and `spark.sql.orc.compression.codec`. If `orc.compress` is given,
- * it overrides `spark.sql.orc.compression.codec`.</li>
- * </ul>
+ * ORC-specific option(s) for writing ORC files can be found in
+ * <a href=
+ * "https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option">
+ * Data Source Option</a> in the version you use.
Review comment:
This is because the options I removed doesn't include generic options.
Would it be better to just add the link to the Generic options no matter what we removed ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842979560
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138661/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842323988
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43151/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842429041
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43155/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842834872
**[Test build #138654 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138654/testReport)** for PR 32546 at commit [`3740c52`](https://github.com/apache/spark/commit/3740c52865a867e7b3e6904a0fefc8304870438b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UpdatingSessionsExec(`
* `class UpdatingSessionsIterator(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-841016889
**[Test build #138548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138548/testReport)** for PR 32546 at commit [`0ce01d0`](https://github.com/apache/spark/commit/0ce01d037a01e45b4b02ba3dbd68dfea940ecaee).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845268840
**[Test build #138750 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138750/testReport)** for PR 32546 at commit [`3971dd8`](https://github.com/apache/spark/commit/3971dd8a95a5d97b25ec8c4224cd7868ec9abe2a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845286974
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138750/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842295245
**[Test build #138632 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138632/testReport)** for PR 32546 at commit [`005d87d`](https://github.com/apache/spark/commit/005d87d34dc6a67b6071852d6179040c686234d9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845026058
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43261/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842814991
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43182/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-844978650
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43264/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845706476
Thanks, @HyukjinKwon .
PR description is updated, and also https://github.com/apache/spark/pull/32204, https://github.com/apache/spark/pull/32161 are updated as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845102642
**[Test build #138738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138738/testReport)** for PR 32546 at commit [`0b0e183`](https://github.com/apache/spark/commit/0b0e183d9e38bcd3f15fef8724ae69ee224f0905).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845112988
**[Test build #138742 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138742/testReport)** for PR 32546 at commit [`6358d59`](https://github.com/apache/spark/commit/6358d5940545021f96f1d9eb52ca3c674d7b01bd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r635796581
##########
File path: docs/sql-data-sources-orc.md
##########
@@ -172,3 +172,32 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC
<td>2.0.0</td>
</tr>
</table>
+
+## Data Source Option
+
+Data source options of ORC can be set via:
+* the `.option`/`.options` methods of `DataFrameReader` or `DataFrameWriter`
+* the `.option`/`.options` methods of `DataStreamReader` or `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>maxFilesPerTrigger</code></td>
Review comment:
This is structured streaming option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845026058
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43261/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32546:
URL: https://github.com/apache/spark/pull/32546
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r635973928
##########
File path: docs/sql-data-sources-orc.md
##########
@@ -172,3 +172,26 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC
<td>2.0.0</td>
</tr>
</table>
+
+## Data Source Option
+
+Data source options of ORC can be set via:
+* the `.option`/`.options` methods of `DataFrameReader` or `DataFrameWriter`
+* the `.option`/`.options` methods of `DataStreamReader` or `DataStreamWriter`
+
+<table class="table">
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>mergeSchema</code></td>
+ <td>None</td>
+ <td>sets whether we should merge schemas collected from all ORC part-files. This will override <code>spark.sql.orc.mergeSchema</code>. The default value is specified in <code>spark.sql.orc.mergeSchema</code>.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>compression</code></td>
+ <td>None</td>
+ <td>compression codec to use when saving to file. This can be one of the known case-insensitive shorten names (none, snappy, zlib, lzo, and zstd). This will override <code>orc.compress</code> and <code>spark.sql.orc.compression.codec</code>. If None is set, it uses the value specified in <code>spark.sql.orc.compression.codec</code>.</td>
+ <td>write</td>
+ </tr>
+</table>
+Other generic options can be found in <a href="https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html"> Generic File Source Options</a>.
Review comment:
If you're going to link generic options here, let's don't mention generic options in the API documentation in the codes as it duplicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-841159278
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842296285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32546:
URL: https://github.com/apache/spark/pull/32546#discussion_r635972738
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
* format("orc").save(path)
* }}}
*
- * You can set the following ORC-specific option(s) for writing ORC files:
- * <ul>
- * <li>`compression` (default is the value specified in `spark.sql.orc.compression.codec`):
- * compression codec to use when saving to file. This can be one of the known case-insensitive
- * shorten names(`none`, `snappy`, `zlib`, `lzo`, and `zstd`). This will override
- * `orc.compress` and `spark.sql.orc.compression.codec`. If `orc.compress` is given,
- * it overrides `spark.sql.orc.compression.codec`.</li>
- * </ul>
+ * ORC-specific option(s) for writing ORC files can be found in
+ * <a href=
+ * "https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option">
+ * Data Source Option</a> in the version you use.
Review comment:
Either way is fine. The point is that we should keep it consistent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842821200
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43182/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-845582125
Looks pretty good otherwise. Don't forgot to update Pr description as well. cc @dongjoon-hyun FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-844943018
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43264/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842555558
**[Test build #138635 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138635/testReport)** for PR 32546 at commit [`aa54b45`](https://github.com/apache/spark/commit/aa54b4542e2fc4c0a5cff9db4ef7da8e7187b3d5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-843250684
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138676/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842295245
**[Test build #138632 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138632/testReport)** for PR 32546 at commit [`005d87d`](https://github.com/apache/spark/commit/005d87d34dc6a67b6071852d6179040c686234d9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-843166651
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43197/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-842776034
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43175/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-843248976
**[Test build #138676 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138676/testReport)** for PR 32546 at commit [`eca1295`](https://github.com/apache/spark/commit/eca1295ef2d82da331c86971b7fbf97374f10753).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-843185107
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43197/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32546:
URL: https://github.com/apache/spark/pull/32546#issuecomment-843064570
**[Test build #138676 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138676/testReport)** for PR 32546 at commit [`eca1295`](https://github.com/apache/spark/commit/eca1295ef2d82da331c86971b7fbf97374f10753).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org