You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/20 03:55:33 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #26958:
[SPARK-30128][SPARK-27627][SPARK-27990][SPARK-11412][FOLLOW-UP]
Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources
'mergeSchema' in ORC
HyukjinKwon opened a new pull request #26958: [SPARK-30128][SPARK-27627][SPARK-27990][SPARK-11412][FOLLOW-UP] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958
### What changes were proposed in this pull request?
This PR adds the options, 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC, into documentation.
- `recursiveFileLookup` at file sources: https://github.com/apache/spark/pull/24830 ([SPARK-27627](https://issues.apache.org/jira/browse/SPARK-27627))
- `pathGlobFilter` at file sources: https://github.com/apache/spark/pull/24518 ([SPARK-27990](https://issues.apache.org/jira/browse/SPARK-27990))
- `mergeSchema` at ORC: https://github.com/apache/spark/pull/24043 ([SPARK-11412](https://issues.apache.org/jira/browse/SPARK-11412))
**Note that** `timeZone` option was not moved from `DataFrameReader.options` as I assume it will likely affect other datasources as well once DSv2 is complete.
### Why are the changes needed?
To document available options in sources properly.
### Does this PR introduce any user-facing change?
In PySpark, `pathGlobFilter` can be set via `DataFrameReader.(text|orc|parquet|json|csv)` and `DataStreamReader.(text|orc|parquet|json|csv)`.
### How was this patch tested?
Manually built the doc and checked the output. Option setting in PySpark is rather a logical change. I manually tested one only:
```bash
$ ls -al tmp
...
-rw-r--r-- 1 hyukjin.kwon staff 3 Dec 20 12:19 aa
-rw-r--r-- 1 hyukjin.kwon staff 3 Dec 20 12:19 ab
-rw-r--r-- 1 hyukjin.kwon staff 3 Dec 20 12:19 ac
-rw-r--r-- 1 hyukjin.kwon staff 3 Dec 20 12:19 cc
```
```python
>>> spark.read.text("tmp", pathGlobFilter="*c").show()
```
```
+-----+
|value|
+-----+
| ac|
| cc|
+-----+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958#issuecomment-567919526
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958#discussion_r360619476
##########
File path: python/pyspark/sql/readwriter.py
##########
@@ -520,20 +537,24 @@ def func(iterator):
raise TypeError("path can be only string, list or RDD")
@since(1.5)
- def orc(self, path, mergeSchema=None, recursiveFileLookup=None):
+ def orc(self, path, mergeSchema=None, pathGlobFilter=None, recursiveFileLookup=None):
"""Loads ORC files, returning the result as a :class:`DataFrame`.
:param mergeSchema: sets whether we should merge schemas collected from all
ORC part-files. This will override ``spark.sql.orc.mergeSchema``.
The default value is specified in ``spark.sql.orc.mergeSchema``.
+ :param pathGlobFilter: an optional glob pattern to only include files with paths matching
+ the pattern. The syntax follows `org.apache.hadoop.fs.GlobFilter`.
+ It does not change the behavior of `partition discovery`_.
:param recursiveFileLookup: recursively scan a directory for files. Using this option
- disables `partition discovery`_.
+ disables `partition discovery`_.
Review comment:
Since we're going ahead for Spark 3, we wont likely backport many things that cause conflicts. So I was thinking it's a feasible option.
Also, I think we might have to document this first that virtical alignment isn't preferred. I think virtical alignment is still valid per PEP 8 and PEP 257.
One downside of doing bit by bit is a confusion by mixed style. Considering that we wont likely add many new docstrings, mixed style exists in a long term.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958#issuecomment-567852680
**[Test build #115615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115615/testReport)** for PR 26958 at commit [`dde76ab`](https://github.com/apache/spark/commit/dde76ab5360a3fb8f3920916f267a7095edc2126).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958#discussion_r360436906
##########
File path: python/pyspark/sql/readwriter.py
##########
@@ -520,20 +537,24 @@ def func(iterator):
raise TypeError("path can be only string, list or RDD")
@since(1.5)
- def orc(self, path, mergeSchema=None, recursiveFileLookup=None):
+ def orc(self, path, mergeSchema=None, pathGlobFilter=None, recursiveFileLookup=None):
"""Loads ORC files, returning the result as a :class:`DataFrame`.
:param mergeSchema: sets whether we should merge schemas collected from all
ORC part-files. This will override ``spark.sql.orc.mergeSchema``.
The default value is specified in ``spark.sql.orc.mergeSchema``.
+ :param pathGlobFilter: an optional glob pattern to only include files with paths matching
+ the pattern. The syntax follows `org.apache.hadoop.fs.GlobFilter`.
+ It does not change the behavior of `partition discovery`_.
:param recursiveFileLookup: recursively scan a directory for files. Using this option
- disables `partition discovery`_.
+ disables `partition discovery`_.
Review comment:
Would you be open to a batch update of all docstrings to tweak the indentation? Or would it be considered as adding too much noise to the git history? I assumed we didn't want to make sweeping changes like that, so my idea was to update the indentation bit by bit as part of more significant changes, which is what I did with `recursiveFileLookup` and `mergeSchema` when I added those in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958#issuecomment-567919531
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115615/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958#issuecomment-567919531
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115615/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958#discussion_r360619514
##########
File path: python/pyspark/sql/readwriter.py
##########
@@ -520,20 +537,24 @@ def func(iterator):
raise TypeError("path can be only string, list or RDD")
@since(1.5)
- def orc(self, path, mergeSchema=None, recursiveFileLookup=None):
+ def orc(self, path, mergeSchema=None, pathGlobFilter=None, recursiveFileLookup=None):
"""Loads ORC files, returning the result as a :class:`DataFrame`.
:param mergeSchema: sets whether we should merge schemas collected from all
ORC part-files. This will override ``spark.sql.orc.mergeSchema``.
The default value is specified in ``spark.sql.orc.mergeSchema``.
+ :param pathGlobFilter: an optional glob pattern to only include files with paths matching
+ the pattern. The syntax follows `org.apache.hadoop.fs.GlobFilter`.
+ It does not change the behavior of `partition discovery`_.
:param recursiveFileLookup: recursively scan a directory for files. Using this option
- disables `partition discovery`_.
+ disables `partition discovery`_.
Review comment:
So .. if you dont mind, I would like this run that separately :-)..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958#issuecomment-567918425
**[Test build #115615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115615/testReport)** for PR 26958 at commit [`dde76ab`](https://github.com/apache/spark/commit/dde76ab5360a3fb8f3920916f267a7095edc2126).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958#issuecomment-568238298
Let me merge this in few days if you guys are fine since it's just a doc change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958#issuecomment-567919526
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26958:
[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and
'pathGlobFilter' in file sources 'mergeSchema' in ORC
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #26958: [SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC
URL: https://github.com/apache/spark/pull/26958#issuecomment-568319384
Thanks all. Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org