You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/16 09:57:20 UTC
[GitHub] [spark] cchighman opened a new pull request #28841: [SPARK-31962] - Provide option to load files after a specified date when reading from a folder path
cchighman opened a new pull request #28841:
URL: https://github.com/apache/spark/pull/28841
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
A new option, _fileModifiedDate_ , is provided expecting a value in 'YYYY-MM-DD HH:mm:ss' format. _InMemoryFileIndex_ considers this option during the process of checking checking for files, just before considering applied _PathFilters_. In order to filter file results, a new PathFilter class was derived for this purpose. General house-keeping around classes extending PathFilter was performed for neatness. It became apparent support was needed to handle multiple potential path filters. Logic was introduced for this purpose and the associated tests written. A new method signature was created in order to maintain backwards compatibility and ensure safety of other features.
This PR presents a very clean way to minimize complexity under various file data source loading scenarios. It's also compatible with structured streaming requiring just a handful of small additions to move forward there. Looking to complete that in a separate PR.
Example Usage: spark.read.format("csv").option("fileModifiedDate","2020-06-15T05:00:00")
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
When loading files from a data source, there can often times be thousands of file within a respective file path. In many cases I've seen, we want to start loading from a folder path and ideally be able to begin loading files having modification dates past a certain point. This would mean out of thousands of potential files, only the ones with modification dates greater than the specified timestamp would be considered. This saves a ton of time automatically and reduces significant complexity managing this in code.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
This PR introduces an option that can be used with Spark file data sources similar to the _latestFirst_ option in structured streaming. An documentation update was made to reflect an example and usage of the new data source option.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
A handful of new unit tests were written and passing. The package was tested locally as well as in a live Databricks environment as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693369156
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r472585442
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PathFilterSuite.scala
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.File
+import java.time.{LocalDateTime, ZoneOffset}
+import java.time.format.DateTimeFormatter
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.util.{stringToFile, CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.test.SharedSparkSession
+
+class PathFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ test("SPARK-31962: when modifiedAfter specified with a past date") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ file.setLastModified(DateTimeUtils.currentTimestamp())
+ val df = spark.read
+ .option("modifiedAfter", "2019-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore specified with a future date") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ val df = spark.read
+ .option("modifiedBefore", "2090-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore specified with a past date") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ file.setLastModified(DateTimeUtils.currentTimestamp())
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedBefore", "1984-05-01T01:00:00")
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with a past date, multiple files, one valid") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file1 = new File(dir, "file1.csv")
+ val file2 = new File(dir, "file2.csv")
+ stringToFile(file1, "text")
+ stringToFile(file2, "text")
+ file1.setLastModified(DateTimeUtils.currentTimestamp())
+ file2.setLastModified(0)
+ val df = spark.read
+ .option("modifiedAfter", "2019-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with a past date, multiple files, both valid") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file1 = new File(dir, "file1.csv")
+ val file2 = new File(dir, "file2.csv")
+ stringToFile(file1, "text")
+ stringToFile(file2, "text")
+ file1.setLastModified(DateTimeUtils.currentTimestamp())
+ file2.setLastModified(DateTimeUtils.currentTimestamp())
+ val df = spark.read
+ .option("modifiedAfter", "2019-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 2)
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with a past date, multiple files, none valid") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file1 = new File(dir, "file1.csv")
+ val file2 = new File(dir, "file2.csv")
+ stringToFile(file1, "text")
+ stringToFile(file2, "text")
+ file1.setLastModified(0)
+ file2.setLastModified(0)
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", "1984-05-01T01:00:00")
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore specified with a future date, " +
+ "multiple files, both valid") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file1 = new File(dir, "file1.csv")
+ val file2 = new File(dir, "file2.csv")
+ stringToFile(file1, "text")
+ stringToFile(file2, "text")
+ file1.setLastModified(0)
+ file2.setLastModified(0)
+
+ val time = LocalDateTime
+ .now()
+ .plusDays(3)
+ .format(DateTimeFormatter
+ .ofPattern("yyyy-MM-dd'T'HH:mm:ss"))
+
+ val df = spark.read
+ .option("modifiedBefore", time)
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 2)
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore specified with a future date, multiple files, one valid") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file1 = new File(dir, "file1.csv")
+ val file2 = new File(dir, "file2.csv")
+ stringToFile(file1, "text")
+ stringToFile(file2, "text")
+
+ file1.setLastModified(0)
+
+ val failTime =
+ LocalDateTime.now().plusDays(3).toEpochSecond(ZoneOffset.UTC)
+ file2.setLastModified(failTime * 1000)
+
+ val time = LocalDateTime
+ .now()
+ .plusHours(10)
+ .format(DateTimeFormatter
+ .ofPattern("yyyy-MM-dd'T'HH:mm:ss"))
+
+ val df = spark.read
+ .option("modifiedBefore", time)
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore specified with a future date, " +
+ "multiple files, none valid") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file1 = new File(dir, "file1.csv")
+ val file2 = new File(dir, "file2.csv")
+ stringToFile(file1, "text")
+ stringToFile(file2, "text")
+
+ val time = LocalDateTime
+ .now()
+ .minusDays(1)
+ .format(DateTimeFormatter
+ .ofPattern("yyyy-MM-dd'T'HH:mm:ss"))
+
+ file1.setLastModified(DateTimeUtils.currentTimestamp())
+ file2.setLastModified(DateTimeUtils.currentTimestamp())
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedBefore", time)
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with a past date and " +
+ "pathGlobalFilter returning results") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ val df = spark.read
+ .option("modifiedAfter", "1984-05-10T01:11:00")
+ .option("pathGlobFilter", "*.csv")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with past date " +
+ "and pathGlobFilter filtering results") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", "1984-05-01T01:00:00")
+ .option("pathGlobFilter", "*.txt")
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with future date and " +
+ "pathGlobFilter returning results") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", "2050-05-01T01:00:00")
+ .option("pathGlobFilter", "*.csv")
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with future date and " +
+ "pathGlobFilter filtering results") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", "2050-05-01T01:00:00")
+ .option("pathGlobFilter", "*.txt")
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore and modifiedAfter are specified out of range and " +
+ "pathGlobFilter returning results") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", "2050-05-01T01:00:00")
+ .option("modifiedBefore", "2050-05-01T01:00:00")
+ .option("pathGlobFilter", "*.csv")
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore and modifiedAfter are specified in range and " +
+ "pathGlobFilter returning results") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ val df = spark.read
+ .option("modifiedAfter", "2019-05-01T01:00:00")
+ .option("modifiedBefore", "2025-05-01T01:00:00")
+ .option("pathGlobFilter", "*.csv")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore and modifiedAfter are specified in range and " +
+ "pathGlobFilter filtering results") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", "2019-05-01T01:00:00")
+ .option("modifiedBefore", "2025-05-01T01:00:00")
+ .option("pathGlobFilter", "*.txt")
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter is specified with an invalid date") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", "2024-05+1 01:00:00")
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(
+ msg.contains("The timestamp provided")
+ && msg.contains("modifiedafter")
+ && msg.contains("2024-05+1 01:00:00"))
Review comment:
nit:
```
Seq("The timestamp provided", "modifiedafter", "2024-05+1 01:00:00").foreach { expectedMsg =>
assert(msg.contains(expectedMsg))
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693369907
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672559298
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663921826
**[Test build #126560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126560/testReport)** for PR 28841 at commit [`5d9a02a`](https://github.com/apache/spark/commit/5d9a02a56c19efa788514fa8225a65d27d7d3a9c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r449241653
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##########
@@ -83,7 +85,7 @@ abstract class PartitioningAwareFileIndex(
val files: Seq[FileStatus] = leafDirToChildrenFiles.get(path) match {
case Some(existingDir) =>
// Directory has children files in it, return them
- existingDir.filter(f => matchGlobPattern(f) && isNonEmptyFile(f))
+ existingDir.filter(f => matchPathPattern(f) && isNonEmptyFile(f))
Review comment:
> `f` here is `FileStatus`. It looks inefficient to get the `FileStatus` again in https://github.com/apache/spark/pull/28841/files#diff-3c816e65d0bbdf16d46c7ea4e0b8cbaeR55
I agree. I considered this in one of my earlier commits actually. There are two places where we handle options in the *FileIndex classes, here in _PartitioningAwareMemoryFileIndex_ and _InMemoryFileIndex_. In both places, we derive from the _PathFilter_ class which exposes an overridable method called _accept_ which takes _Path_ as its only parameter.
I could break with the abstraction and overload the method so that it would take a FileStatus directly. I'll look into this unless there are any objections.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693202858
**[Test build #128746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128746/testReport)** for PR 28841 at commit [`1c8384c`](https://github.com/apache/spark/commit/1c8384cebe05fbc1ebb3c86660e57508b031c6f9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-675297983
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721761116
**[Test build #130606 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130606/testReport)** for PR 28841 at commit [`aed5d30`](https://github.com/apache/spark/commit/aed5d3062c05e8cd584ae56d33259125b76ba6d1).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693227460
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-652894802
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672557271
**[Test build #127358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127358/testReport)** for PR 28841 at commit [`cf9d2fa`](https://github.com/apache/spark/commit/cf9d2fa70c88ece8f377c12967c84602f56cc9b4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-664140348
sparkr passed this time too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-664121661
**[Test build #126611 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126611/testReport)** for PR 28841 at commit [`b7952a6`](https://github.com/apache/spark/commit/b7952a6af9694354972d8cb4cfc0548e6cecc1f8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693588238
**[Test build #128766 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128766/testReport)** for PR 28841 at commit [`1c8384c`](https://github.com/apache/spark/commit/1c8384cebe05fbc1ebb3c86660e57508b031c6f9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ModifiedBeforeFilter(thresholdTime: Long, val timeZoneId: String)`
* `class ModifiedAfterFilter(thresholdTime: Long, val timeZoneId: String)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663921826
**[Test build #126560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126560/testReport)** for PR 28841 at commit [`5d9a02a`](https://github.com/apache/spark/commit/5d9a02a56c19efa788514fa8225a65d27d7d3a9c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663975313
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672086902
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-695155611
@gengliangwang @cloud-fan @maropu @Dooyoung-Hwang @HeartSaVioR @HyukjinKwon
Following up on this PR for next steps. We've seen a strong amount of constructive feedback during the review process but haven't convinced anyone yet to approve.
I went to revisit the commit log in efforts to identify other potential reviewers with expertise in our file.. I found our group represented quite well here with @gengliangwang early on suggesting to follow an existing method for consistency.
Comments?
Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672093140
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127330/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-664701314
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-707389675
ping @cchighman
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r472591099
##########
File path: docs/sql-data-sources-generic-options.md
##########
@@ -119,3 +119,48 @@ To load all files recursively, you can use:
{% include_example recursive_file_lookup r/RSparkSQLExample.R %}
</div>
</div>
+
+### Modification Time Path Filters
+`modifiedBefore` and `modifiedAfter` are options that can be
+applied together or separately in order to achieve greater
+granularity over which files may load during a Spark batch query.
+
+When the `timeZone` option is present, modified timestamps will be
+interpreted according to the specified zone. When a timezone option
+is not provided, modified timestamps will be interpreted according
+to the default zone specified within the Spark configuration. Without
+any timezone configuration, modified timestamps are interpreted as UTC.
+
+`modifiedBefore` will only allow files having last modified
+timestamps occurring before the specified time to load. For example,
+when`modifiedBefore` has the timestamp `2020-06-01T12:00:00` applied,
+all files modified after that time will not be considered when loading
+from a file data source.
+
+`modifiedAfter` only allows files having last modified timestamps
+occurring after the specified timestamp. For example, when`modifiedAfter`
+has the timestamp `2020-06-01T12:00:00` applied, only files modified after
+this time will be eligible when loading from a file data source. When both
+`modifiedBefore` and `modifiedAfter` are specified together, files having
+last modified timestamps within the resulting time range are the only files
+allowed to load.
Review comment:
How about rephrasing it like this?
```
* `modifiedBefore`: an optional timestamp to only include files with
modification times occurring before the specified time. The provided timestamp
must be in the following format: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)
* `modifiedAfter`: an optional timestamp to only include files with
modification times occurring after the specified time. The provided timestamp
must be in the following format: YYYY-MM-DDTHH:mm:ss (e.g. 2020-06-01T13:00:00)
Note that, when the `timeZone` option is present, `modifiedBefore`/`modifiedAfter`
will be interpreted according to the specified zone. When a timezone option
is not provided, the timestamps will be interpreted according
to the default zone specified within the Spark configuration. Without
any timezone configuration, modified timestamps are interpreted as UTC.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-666133472
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-653059192
**[Test build #124887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124887/testReport)** for PR 28841 at commit [`150e77d`](https://github.com/apache/spark/commit/150e77db17957d1776d88cbd82722c0791c6b583).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721731734
**[Test build #130600 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130600/testReport)** for PR 28841 at commit [`4480ca7`](https://github.com/apache/spark/commit/4480ca7bfbf8a5bce19ab9c3c364d5866e3ea135).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-687627744
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-695161492
> Shall we use `modificationTimeFilter`? the word `date` is not accurate since it can be referred to date without any hour/minute/seconds
Semantics for the new options were thoughtfully considered. @gengliangwang comment here led to realizing that descriptions of this feature in terms of date did not accurately convey it's purpose. As a result, everything was adjusted to "time".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721747360
**[Test build #130604 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130604/testReport)** for PR 28841 at commit [`1d54fa5`](https://github.com/apache/spark/commit/1d54fa55c08b12919b1e94f31cf0154ded5133f2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672562791
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r449243486
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##########
@@ -83,7 +85,7 @@ abstract class PartitioningAwareFileIndex(
val files: Seq[FileStatus] = leafDirToChildrenFiles.get(path) match {
case Some(existingDir) =>
// Directory has children files in it, return them
- existingDir.filter(f => matchGlobPattern(f) && isNonEmptyFile(f))
+ existingDir.filter(f => matchPathPattern(f) && isNonEmptyFile(f))
Review comment:
I can put a wrapper around this to make the behavior more efficient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-679882932
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-653966315
I still feel the date option can be ranged via two symmetric options; while this won't help on SS case, this can help filtering out files in the range of date which are out of interest - while this patch only filters on the left side, we can do that for the right side as well. There was similar review comment from other contributor.
For `filesModifiedBefore` vs `fromModifiedDate`, I think former sounds more natural and even `modifiedBefore` sounds enough (as we know it deals with files), but let's hear others' opinions as well.
Before looking at the code diff, some points on PR description (note that PR title and description become commit message):
> It's also compatible with structured streaming requiring just a handful of small additions to move forward there. Looking to complete that in a separate PR.
The part is better to be just left to PR comment rather than the part of the commit message. Let's drop it from there.
> This PR introduces an option that can be used with Spark file data sources similar to the latestFirst option in structured streaming.
latestFirst option doesn't work like this, as you may already know. (If latestFirst works like this I won't have a headache.) It'd be better to explain the new option there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-653932007
**[Test build #124948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124948/testReport)** for PR 28841 at commit [`eb8b3e6`](https://github.com/apache/spark/commit/eb8b3e647c85730e567971bed26fd67cc303fdc7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r483940721
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PathFilterSuite.scala
##########
@@ -0,0 +1,553 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.File
+import java.time.{LocalDateTime, ZoneOffset}
+import java.time.format.DateTimeFormatter
+
+import scala.util.Random
+
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.util.{stringToFile, DateTimeUtils}
+import org.apache.spark.sql.test.SharedSparkSession
+
+
+class PathFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ def createSingleFile(dir: File): File = {
+ val file = new File(dir, "temp" + Random.nextInt(1000) + ".csv")
+ stringToFile(file, "text")
+ }
+
+ def setFileTime(time: LocalDateTime, file: File): Boolean = {
+ val sameTime = time.toEpochSecond(ZoneOffset.UTC)
+ file.setLastModified(sameTime * 1000)
+ }
+
+ def setPlusFileTime(time: LocalDateTime, file: File, interval: Long): Boolean = {
+ val sameTime = time.plusDays(interval).toEpochSecond(ZoneOffset.UTC)
+ file.setLastModified(sameTime * 1000)
+ }
+
+ def setMinusFileTime(time: LocalDateTime, file: File, interval: Long): Boolean = {
+ val sameTime = time.minusDays(interval).toEpochSecond(ZoneOffset.UTC)
+ file.setLastModified(sameTime * 1000)
+ }
+
+ def formatTime(time: LocalDateTime): String = {
+ time.format(DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss"))
+ }
+
+ test("SPARK-31962: when modifiedBefore specified" +
+ " and sharing same timestamp with file last modified time.") {
+ withTempDir { dir =>
+ val file = createSingleFile(dir)
+ val time = LocalDateTime.now(ZoneOffset.UTC)
+ setFileTime(time, file)
+ val formattedTime = formatTime(time)
+
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedBefore", formattedTime)
Review comment:
@maropu
I updated this source file after running the scalafmt maven task. Attached is the output based on our current config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650052008
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-652843996
@HeartSaVioR I still think implementing this at the _PartitioningAwareFileIndex_ level makes a lot of sense and bypasses all the complexities you mentioned above. There can be some cases where you begin streaming from a file source that could have hundreds of thousands of files and many with the same timestamp. You want to start the process at a specified point. _PartitioningAwareFileIndex_ is processed before any other options for structured streaming are considered during _fetchMaxOffset_. I believe _modifiedDateFilter_ is a great way to determine where you want to start streaming from and is limited to that use case. The semantics for offset I believe completely apply but I think they would apply to the files that are returned from _InMemoryFileIndex_ or _MetadataLogFileIndex_.
This option is very intuitive for the consumer because, for any given path, they can explicitly set the population of files that would be considered for structured streaming. `allFiles` in `fetchMaxOffset` would return the starting point that would represent the earliest/latest offsets. Do you see the difference?
Granted, I can conceptualize how this could be implemented in _FileStreamSource_. It seems though like the problems you're describing shouldn't impact how we would ultimately filter files based on parameters which seek to limit more of an unbounded problem we might have currently? I'm asking this just to understand if the complexity is as easy as just adding an extra layer of filtering if the options are specified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-722330354
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r472592405
##########
File path: docs/sql-data-sources-generic-options.md
##########
@@ -119,3 +119,48 @@ To load all files recursively, you can use:
{% include_example recursive_file_lookup r/RSparkSQLExample.R %}
</div>
</div>
+
+### Modification Time Path Filters
+`modifiedBefore` and `modifiedAfter` are options that can be
+applied together or separately in order to achieve greater
+granularity over which files may load during a Spark batch query.
+
+When the `timeZone` option is present, modified timestamps will be
+interpreted according to the specified zone. When a timezone option
+is not provided, modified timestamps will be interpreted according
+to the default zone specified within the Spark configuration. Without
+any timezone configuration, modified timestamps are interpreted as UTC.
+
+`modifiedBefore` will only allow files having last modified
+timestamps occurring before the specified time to load. For example,
+when`modifiedBefore` has the timestamp `2020-06-01T12:00:00` applied,
+all files modified after that time will not be considered when loading
+from a file data source.
+
+`modifiedAfter` only allows files having last modified timestamps
+occurring after the specified timestamp. For example, when`modifiedAfter`
+has the timestamp `2020-06-01T12:00:00` applied, only files modified after
+this time will be eligible when loading from a file data source. When both
+`modifiedBefore` and `modifiedAfter` are specified together, files having
+last modified timestamps within the resulting time range are the only files
+allowed to load.
Review comment:
> Note that, when the `timeZone` option is present, `modifiedBefore`/`modifiedAfter`
will be interpreted according to the specified zone. When a timezone option
is not provided, the timestamps will be interpreted according
to the default zone specified within the Spark configuration. Without
any timezone configuration, modified timestamps are interpreted as UTC.
We could say it like this?
```
When a timezone option is not provided, the timestamps will be interpreted according
to the Spark session timezone (`spark.sql.session.timeZone`).
```
https://github.com/apache/spark/pull/28841/files#diff-2614a5c9164a734b0208806133aa7de9R85-R88
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721759776
**[Test build #130606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130606/testReport)** for PR 28841 at commit [`aed5d30`](https://github.com/apache/spark/commit/aed5d3062c05e8cd584ae56d33259125b76ba6d1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r472595471
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PathFilterSuite.scala
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.File
+import java.time.{LocalDateTime, ZoneOffset}
+import java.time.format.DateTimeFormatter
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.util.{stringToFile, CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.test.SharedSparkSession
+
+class PathFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ test("SPARK-31962: when modifiedAfter specified with a past date") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ file.setLastModified(DateTimeUtils.currentTimestamp())
+ val df = spark.read
+ .option("modifiedAfter", "2019-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore specified with a future date") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ val df = spark.read
+ .option("modifiedBefore", "2090-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore specified with a past date") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ file.setLastModified(DateTimeUtils.currentTimestamp())
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedBefore", "1984-05-01T01:00:00")
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with a past date, multiple files, one valid") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file1 = new File(dir, "file1.csv")
+ val file2 = new File(dir, "file2.csv")
+ stringToFile(file1, "text")
+ stringToFile(file2, "text")
+ file1.setLastModified(DateTimeUtils.currentTimestamp())
Review comment:
These timestamps should be inclusive or exclusive? e.g., `(modifiedAfter, modifiedBefore)`, `[modifiedAfter, modifiedBefore]`... Looks like the current code follows `(modifiedAfter, modifiedBefore]`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721819575
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672156463
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673188200
**[Test build #127391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127391/testReport)** for PR 28841 at commit [`74b4d33`](https://github.com/apache/spark/commit/74b4d3378ec9b4021631aa2f46ed002e9a938f52).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r468636017
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/pathfilters/FileIndexFilter.scala
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.pathfilters
+
+import org.apache.hadoop.fs.{FileStatus, PathFilter}
+
+
+trait FileIndexFilter extends PathFilter with Serializable {
+ def accept(fileStatus: FileStatus): Boolean
+ def strategy(): String
Review comment:
There is a direct string comparison between the option specified and the string representation of the strategy in PathFilterStrategies. That's why its a string. Its not meant to represent a class per say.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721731734
**[Test build #130600 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130600/testReport)** for PR 28841 at commit [`4480ca7`](https://github.com/apache/spark/commit/4480ca7bfbf8a5bce19ab9c3c364d5866e3ea135).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650558195
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r468993324
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/pathFilters.scala
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.util.{Locale, TimeZone}
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, GlobFilter, Path, PathFilter}
+
+import org.apache.spark.sql.{AnalysisException, SparkSession}
+import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.execution.datasources.pathfilters.PathFilterStrategy
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.unsafe.types.UTF8String
+
+
+/**
+ * SPARK-31962: Provide modifiedAfter and modifiedBefore options when
+ * filtering from a batch-based file data source.
+ *
+ * Example Usages
+ * Load all CSV files modified after date:
+ * spark.read.format("csv").option("modifiedAfter","2020-06-15T05:00:00").load()
+ *
+ * Load all CSV files modified before date:
+ * spark.read.format("csv").option("modifiedBefore","2020-06-15T05:00:00").load()
+ *
+ * Load all CSV files modified between two dates:
+ * spark.read.format("csv").option("modifiedAfter","2019-01-15T05:00:00")
+ * .option("modifiedBefore","2020-06-15T05:00:00").load()
+ *
+@param sparkSession SparkSession
+@param hadoopConf Hadoop Configuration object
+@param options Map containing options
+ */
+abstract class ModifiedDateFilter(sparkSession: SparkSession,
+ hadoopConf: Configuration,
+ options: CaseInsensitiveMap[String])
+ extends PathFilterStrategy(sparkSession, hadoopConf, options) {
+ lazy val timeZoneId: String = options.getOrElse(
+ DateTimeUtils.TIMEZONE_OPTION.toLowerCase(Locale.ROOT),
+ SQLConf.get.sessionLocalTimeZone)
+
+ /* Implicitly defaults to UTC if unable to parse */
+ lazy val timeZone: TimeZone = DateTimeUtils.getTimeZone(timeZoneId)
+ lazy val timeString: UTF8String =
+ UTF8String.fromString(options.apply(strategy()))
+
+ def thresholdTime(): Long = {
+ DateTimeUtils
+ .stringToTimestamp(timeString, timeZone.toZoneId)
+ .getOrElse(throw new AnalysisException(
+ s"The timestamp provided for the '${strategy()}'" +
+ s" option is invalid. The expected format is 'YYYY-MM-DDTHH:mm:ss'. " +
+ s" Provided timestamp: " +
+ s"${options.apply(strategy())}"))
+ }
+
+ def localTime(micros: Long): Long =
+ DateTimeUtils.fromUTCTime(micros, timeZoneId)
+
+ def accept(fileStatus: FileStatus): Boolean
+ def accept(path: Path): Boolean
+ def strategy(): String
+}
+
+/**
+ * Filter used to determine whether file was modified
+ * before the provided timestamp.
+ *
+ @param sparkSession SparkSession
+ @param hadoopConf Hadoop Configuration object
+ @param options Map containing options
+ */
+class ModifiedBeforeFilter(sparkSession: SparkSession,
+ hadoopConf: Configuration,
+ options: CaseInsensitiveMap[String])
+ extends ModifiedDateFilter(sparkSession, hadoopConf, options)
+ with FileIndexFilter {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673191372
**[Test build #127391 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127391/testReport)** for PR 28841 at commit [`74b4d33`](https://github.com/apache/spark/commit/74b4d3378ec9b4021631aa2f46ed002e9a938f52).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-679624551
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721782501
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-644667095
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650083552
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-679882505
**[Test build #127883 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127883/testReport)** for PR 28841 at commit [`263dd2a`](https://github.com/apache/spark/commit/263dd2a58ee990600aae3c40ea3eb56368a9c48d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-687627544
**[Test build #128319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128319/testReport)** for PR 28841 at commit [`d57bf08`](https://github.com/apache/spark/commit/d57bf088bd581bd5cd1c1ea81f44581a1f20a6ed).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650037305
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-687671184
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-679879391
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721748687
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130604/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721739507
**[Test build #130602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130602/testReport)** for PR 28841 at commit [`40c99dc`](https://github.com/apache/spark/commit/40c99dccb659b56f531144dcbc57598e8914e4ba).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650898695
> I wonder though if structured streaming always implied an event source, particularly when streaming from a file source?
Ideally it should be. It's not 100% really for file stream source (as it doesn't store offset and allows reading late files), but it's partially considered as file stream source "sorts" the files by its timestamp, in both directions, forward and backward. Probably you're thinking about "offset" as a file offset, while I mean "offset" to mark where we read so far. In this case, modified time, with some processed files as there could be new files having same modified time (some of files may not be picked up due to max number of files per trigger).
> For example, modifiedDateFilter applies specifically to a point in time when you begin structured streaming on a file data source. You would not have an offset yet in commitedOffsets. The offset use case would imply you are restarting an existing, checkpointed stream or attempting to read from a checkpointed location, right?
Yes, but you'll want to consider the case where end users "changes" modifiedDataFilter before restaring the query from the checkpoint. Every options can be changed during restart, which makes things very complicated. You'll need to consider combinations of `modifiedDataFilter`, `latestFirst`, `maxFilesAge`. And also make sure the behavior is intuitive to the end users. Please refer #28422 for what I proposed and which review comments it got.
> When using an offset with a Kafka data source, some write has occurred by which a written checkpoint exists. With the file data source for files that have not yet been read or written, I'm curious how I would apply offset bounds in this way. I was thinking I would have to be reading from a data source that had used structured streaming with checkpointing in order for the offset to exist (committed).
>
> Does this make sense? It seems like once you've written a checkpoint while writing to a stream from the readStream dataframe that's loading files, you would have clear context to apply offset-based semantics.
Yeah, that should be ideally an "offset" instead of file stream source's own metadata, but I'm not sure how far we want to go. Probably I've been thinking too far, but there's a real issue on file stream source metadata growing, and I support the way it also clearly fixes the issue. I'm worried that once we introduce a relevent option without enough consideration, that would bug us via respecting backward compatiblity.
> With_startingOffsetByTimestamp_, you have the ability to indicate start/end offsets per topic such as TopicA or TopicB. If this concept were applied to a file data source with the underlying intent that each file name represented a topic, problems begin to emerge. For example, if there are multiple files, they would have different file names, different file names may imply a new topic.
`underlying intent that each file name represented a topic` - I don't think so (If you thought I proposed it then I didn't mean that). Do you remember a file is an unit of processing on file source? Spark doesn't process the file partially, in other words, Spark never splits the file across micro-batches. If you reflect the file source as Kafka data source, an record in Kafka data source = a file in File data source.
So the timestamp can represent the offset how Spark has been read so far - though there're some issues below:
1) We allowed to read late files - there's new updated file but modified time being set to "previous time". I don't think it should be, but current File data source allows to do so, so that's arguable topic.
2) There can be lots of files in same timestamp, and Spark has max files to process in a batch. That means there's a case only partial files get processed. That's why only storing timestamp doesn't work here and we still need to store some processed files as well to mitigate.
3) That's technically really odd, but considering eventual consistency like S3... Listing operation may not show some of available files. This may be a blocker to use timestamp as an offset, but even with eventual consistency we don't expect it shows after days. (Do I understand correctly on expectation for eventual consistency on S3?) That said, (ts for (timestamp - some margin) + processed files within margin) could be used for timestamp instead. We can leverage purge timestamp in SeenFileMap, though I think 7 days are too long. And latestFirst should be replaced with something else, because it just simply defeats maxFilesAge and unable to apply offset-like approach.
Does it make sense?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-646396659
**[Test build #124247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124247/testReport)** for PR 28841 at commit [`256cb1b`](https://github.com/apache/spark/commit/256cb1bfe717c9cf0ce2e4204125295ae7569895).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650037305
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-664116363
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-652807186
@HeartSaVioR Thank you for your detailed comments. I've been digging into the PR you mentioned along with the associated Kafka Batch sources, etc. I'm leaning towards separating the PRs mainly to reduce complexity in any one PR. I have a few questions.
1.) By separating these PRs, the offset-based semantics would just apply to structured streaming correct? Meaning, _modifiedDateFilter_ would just be used for the batch case? The Kafka batch example uses batch reading with offset-based semantics but that seems unintuitive for the file data source uses case.
2.) _startingOffsetbyTimestamp_ and the associated semantics refer to _the start point of timestamp when a query is started_. In the file stream source use case, there seems to be a distinctive difference between the file _modified date_ and when the query itself is started. From what I'm gathering, because an offset represents a file itself, the language in this sense would actually relate the the modified timestamp on the file as opposed to when the query itself was started? In effect, the file stream is abstract based on the modified time of the file itself?
3.) If a file is modified and exists in SeenFilesMap, but is subsequently modified, I'm guessing one file being modified means the entire file will be reconsumed as we don't consider partial files, correct?
4.) Is there an ideal way to exclude the streaming use case from _PartioningAwareFileIndex_?
Thank you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman edited a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman edited a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-647838170
> @gengliangwang @dongjoon-hyun
>
> Are there any strong feelings against making `pathGlobfilter` a derived class strategy, along with the disjointed implementation in `CommandUtils` for `PathIgnoreNonData`, as been implemented here in `InMemoryFileIndex` except that the implementation approach and strategy would actually be according to `pathGlobFilter` in `PartitioningAwareFileIndex`?
>
> In effect, the implementation in my change above in `listLeafFiles` was extending what appears to be a duplication of the initial strategy used here for `pathGlobFilter`. In `PartitioningAwareFileIndex`, the `allFiles` method uses the same filtering methodology.
>
> So, the difference would be the below method signature taking the same strategy approach on the abstraction to determine which filters to apply:
>
> ` protected def matchGlobPattern(file: FileStatus): Boolean = { pathGlobFilter.forall(_.accept(file.getPath)) }`
>
> _changes to_
>
> ` protected def matchPathFilter(file: FileStatus): Boolean = { pathFilters.forall(_.accept(file.getPath)) }`
>
> This would mean there is little implementation changes in the current code and it can be easily extended. `GlobFilter`, being a derived class from `PathFilter`, makes this a rather perfect fit since each case is implementing the `accept` interface. We would also be consolidating these two different filtering approaches in the same place, reducing code, as to eliminate any sort of confusion in the future.
>
> I can't see any discernable benefit as to why `InMemoryFileIndex` needs a separate implementation in `listLeafFiles` for path filtering that isn't already handled when filtering during `allfiles` where `pathGlobFilter` is implemented.
> @gengliangwang @dongjoon-hyun
>
> Are there any strong feelings against making `pathGlobfilter` a derived class strategy, along with the disjointed implementation in `CommandUtils` for `PathIgnoreNonData`, as been implemented here in `InMemoryFileIndex` except that the implementation approach and strategy would actually be according to `pathGlobFilter` in `PartitioningAwareFileIndex`?
>
> In effect, the implementation in my change above in `listLeafFiles` was extending what appears to be a duplication of the initial strategy used here for `pathGlobFilter`. In `PartitioningAwareFileIndex`, the `allFiles` method uses the same filtering methodology.
>
> So, the difference would be the below method signature taking the same strategy approach on the abstraction to determine which filters to apply:
>
> ` protected def matchGlobPattern(file: FileStatus): Boolean = { pathGlobFilter.forall(_.accept(file.getPath)) }`
>
> _changes to_
>
> ` protected def matchPathFilter(file: FileStatus): Boolean = { pathFilters.forall(_.accept(file.getPath)) }`
>
> This would mean there is little implementation changes in the current code and it can be easily extended. `GlobFilter`, being a derived class from `PathFilter`, makes this a rather perfect fit since each case is implementing the `accept` interface. We would also be consolidating these two different filtering approaches in the same place, reducing code, as to eliminate any sort of confusion in the future.
>
> I can't see any discernable benefit as to why `InMemoryFileIndex` needs a separate implementation in `listLeafFiles` for path filtering that isn't already handled when filtering during `allfiles` where `pathGlobFilter` is implemented.
The need arises since we only have one filtering strategy where `pathGlobFilter` is used in `PartitioningAwareFileIndex` and handling cleanly if we now have more than one approach to path filtering so the implementation I'm suggesting would be nearly identical to above except moving it to where `pathGlobFilter` is implemented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-664140210
Hmm...doubt this is related. Doesn't happen locally either.
```
======================================================================
FAIL: test_train_prediction (pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests)
Test that error on test data improves as model is trained.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/spark/spark/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 466, in test_train_prediction
eventually(condition, timeout=180.0)
File "/home/runner/work/spark/spark/python/pyspark/testing/utils.py", line 81, in eventually
lastValue = condition()
File "/home/runner/work/spark/spark/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 461, in condition
self.assertGreater(errors[1] - errors[-1], 2)
AssertionError: 1.672640157855923 not greater than 2
----------------------------------------------------------------------
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673191390
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-671985182
**[Test build #127330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127330/testReport)** for PR 28841 at commit [`e6f8928`](https://github.com/apache/spark/commit/e6f89283b1b839d7e5071c87deec48893865dec8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672557271
**[Test build #127358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127358/testReport)** for PR 28841 at commit [`cf9d2fa`](https://github.com/apache/spark/commit/cf9d2fa70c88ece8f377c12967c84602f56cc9b4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672660042
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663848164
**[Test build #126544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126544/testReport)** for PR 28841 at commit [`853c432`](https://github.com/apache/spark/commit/853c4324958e0b756d51b6d9b5c94e17fb6ebb7a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693370587
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-671989831
**[Test build #127331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127331/testReport)** for PR 28841 at commit [`042c36e`](https://github.com/apache/spark/commit/042c36ed531b0d6e76295b5e9b17889f036a160f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672095098
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127339/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-725767515
@maropu
Thank you for your feedback. I will finish the merge pieces this evening. If one of you two would like to pick up any remaining effort, please feel free to do so.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663848281
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-664701314
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-647880453
> The need arises since we only have one filtering strategy where pathGlobFilter is used in PartitioningAwareFileIndex and handling cleanly if we now have more than one approach to path filtering so the implementation I'm suggesting would be nearly identical to above except moving it to where pathGlobFilter is implemented.
@cchighman the option should be effective for both `MetadataLogFileIndex` and `InMemoryFileIndex`. That's why I suggest implementing it in `PartitioningAwareFileIndex`. I think we can create a method to determine which files are matched according to the parameters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693200082
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman edited a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman edited a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-668096119
> Anyway, thanks for the valable work, @cchighman !
@maropu
I appreciate all your feedback and will address each comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-664176747
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-671985860
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-647012367
**[Test build #124326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124326/testReport)** for PR 28841 at commit [`918af78`](https://github.com/apache/spark/commit/918af78219225dd130a36d539212659fc9c9df8c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r441280950
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
##########
@@ -19,18 +19,18 @@ package org.apache.spark.sql.execution.datasources
import java.io.{File, FileNotFoundException}
Review comment:
It just occurred to me my styling, while passing lint, is a bit of noise here. I will push a correction as to reflect only meaningful changes.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
##########
@@ -176,9 +180,27 @@ object InMemoryFileIndex extends Logging {
private[sql] def bulkListLeafFiles(
paths: Seq[Path],
hadoopConf: Configuration,
- filter: PathFilter,
+ filters: ParArray[PathFilter],
sparkSession: SparkSession,
areRootPaths: Boolean): Seq[(Path, Seq[FileStatus])] = {
+ bulkListLeafFiles(paths, hadoopConf, filters, sparkSession, areRootPaths, Map[String, String]())
Review comment:
For all other classes that leverage InMemoryFileIndex but are not prepared to pass along any options, the above method is an adapter to add an initialized map for the options parameter at the end. In the implementation, an empty map will just be passed by.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
##########
@@ -295,14 +319,14 @@ object InMemoryFileIndex extends Logging {
*
* @return all children of path that match the specified filter.
*/
- private def listLeafFiles(
- path: Path,
+ private def listLeafFiles(path: Path,
hadoopConf: Configuration,
- filter: PathFilter,
+ filters: ParArray[PathFilter],
Review comment:
Given how _listLeafFiles_ is right at the center of building file indexes to return back to a caller, there are lots of opportunities here for consumers to require more than one filter / configuration / option. Adding support for any number of filters as opposed to just one filter feels like a good pay off.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
##########
@@ -356,26 +380,35 @@ object InMemoryFileIndex extends Logging {
bulkListLeafFiles(
dirs.map(_.getPath),
hadoopConf,
- filter,
+ filters,
session,
- areRootPaths = false
+ areRootPaths = false,
+ parameters = parameters
).flatMap(_._2)
case _ =>
dirs.flatMap { dir =>
listLeafFiles(
dir.getPath,
hadoopConf,
- filter,
+ filters,
sessionOpt,
ignoreMissingFiles = ignoreMissingFiles,
ignoreLocality = ignoreLocality,
- isRootPath = false)
+ isRootPath = false,
+ parameters = parameters)
}
}
+ val fileFilters = (filters ++ FileModifiedDateOption.accept(
Review comment:
Followed a similar methodlogy as FileStreamOptions or other such classes that aim to deserialize some condition and perform a resulting action. This helps keep everything clean.
##########
File path: docs/sql-data-sources-generic-options.md
##########
@@ -119,3 +119,14 @@ To load all files recursively, you can use:
{% include_example recursive_file_lookup r/RSparkSQLExample.R %}
</div>
</div>
+
+### Load Files after Modification Date
+`fileModifiedDate` is an option used to only load files after a specified modification date.
Review comment:
I took a guess here on best place and way to describe the new file data source option. Structured Streaming had a well defined layout to document options such as latestFirst and maxFileAge.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/pathfilters/FileModifiedDateOption.scala
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.pathfilters
+
+import java.time.{LocalDateTime, ZoneOffset}
+import java.time.format.DateTimeFormatter
+
+import scala.collection.parallel.mutable.ParArray
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.PathFilter
+
+import org.apache.spark.sql.SparkSession
+
+/**
+SPARK-31962 - Provide option to load files after a specified
+date when reading from a folder path.
+ */
+object FileModifiedDateOption {
+ def accept(parameters: Map[String, String],
+ sparkSession: SparkSession,
+ hadoopConf: Configuration): ParArray[PathFilter] = {
+ var afterDateSeconds = 0L
+ val option = "fileModifiedDate"
+ val filesModifiedAfterDate = parameters.get(option)
+ val hasModifiedDateOption = filesModifiedAfterDate.isDefined
+
+ if (hasModifiedDateOption) {
+ filesModifiedAfterDate.foreach(fileDate => {
+ val formatter = DateTimeFormatter.ISO_LOCAL_DATE_TIME
+ val locateDate = LocalDateTime.parse(fileDate, formatter)
+ afterDateSeconds = locateDate.toEpochSecond(ZoneOffset.UTC) * 1000
+ })
+ val fileFilter =
+ new PathFilterIgnoreOldFiles(sparkSession, hadoopConf, afterDateSeconds)
+
+ return ParArray[PathFilter](fileFilter)
+ }
+ ParArray[PathFilter]()
Review comment:
If the parameter isn't specified at all, the check on line 43 evaluates to false and we return back an empty object.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
##########
@@ -356,26 +380,35 @@ object InMemoryFileIndex extends Logging {
bulkListLeafFiles(
dirs.map(_.getPath),
hadoopConf,
- filter,
+ filters,
session,
- areRootPaths = false
+ areRootPaths = false,
+ parameters = parameters
).flatMap(_._2)
case _ =>
dirs.flatMap { dir =>
listLeafFiles(
dir.getPath,
hadoopConf,
- filter,
+ filters,
sessionOpt,
ignoreMissingFiles = ignoreMissingFiles,
ignoreLocality = ignoreLocality,
- isRootPath = false)
+ isRootPath = false,
+ parameters = parameters)
}
}
+ val fileFilters = (filters ++ FileModifiedDateOption.accept(
+ parameters,
+ sessionOpt.get,
+ hadoopConf))
+ .filter(filter => filter !=null)
+
val allFiles = topLevelFiles ++ nestedFiles
- if (filter != null) allFiles.filter(f => filter.accept(f.getPath)) else allFiles
+ if (fileFilters.nonEmpty) allFiles.filter(file => (
+ fileFilters.forall(filter => filter != null && filter.accept(file.getPath))))
Review comment:
Outside of a class for testing, there was only one other PathFilter extended class that was inline to CommandUtils.scala as seen in this PR. After adding my own for handling the modified date, I placed both classes in the spark.sql.execution.datasources.pathfilters folder.
When considering the above code on line 408, if filters exist in the collection, we then want to see if we should filter any resulting files from _allFiles_ before returning back to the caller. The second lambda, _fileFilters.forall_, provides a collective boolean result based upon how an accept method for a given filter decides to respond.
Since a ParArray collection is used to parallelize invocation of each field, there wasnt any noticeable performance impact.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
##########
@@ -34,20 +35,10 @@ import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.util.{ArrayData, GenericArrayData}
import org.apache.spark.sql.execution.QueryExecution
import org.apache.spark.sql.execution.datasources.{DataSourceUtils, InMemoryFileIndex}
+import org.apache.spark.sql.execution.datasources.pathfilters.PathFilterIgnoreNonData
import org.apache.spark.sql.internal.{SessionState, SQLConf}
import org.apache.spark.sql.types._
-/**
- * For the purpose of calculating total directory sizes, use this filter to
- * ignore some irrelevant files.
- * @param stagingDir hive staging dir
- */
-class PathFilterIgnoreNonData(stagingDir: String) extends PathFilter with Serializable {
Review comment:
The pattern used here by a previous committer is pretty nice. I actually wrote this a different way until I realized a Hadoop.fs.Path can be a file. Currently in both file data source and file stream source, we don't have much consistency around where or how we check for options that may filter results.
That's what makes this interface implementation for PathFilter pretty cool. You can encapsulate these option implementations into visitor classes and during the normal process of InMemoryFileIndex execution, service both styles with less code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650086619
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman edited a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman edited a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-644793641
Thanks for your comments, @bart-samwel. I like your way of thinking, there are a lot of unique cases here. To provide more context behind the scenario I'm looking to cover which is a current issue for consumers:
- Imagine you have a massive, massive data lake with routine ETL operations.
- Every couple hours or so, a CSV file is dropped in a "Delta" folder containing perhaps 50 million events, per dataset, and you have a lot of these various datasets.
- Over time, going back a handful of years, the folder hierarchy was rather deterministic which seems to be a common practice, such that you have _/dataset/delta/yyyy-mm-dd/dataset_guid_timestamp.csv_ as folder structure.
- A number of teams may need to begin consuming these files but they are only interested in consuming them starting from a particular date. Prior to this date, there is no longer any interest, and they hope to consume all the delta files for events up to the current date from the specified modified date without needing to write code that concatenates or embeds this for them.
- From this perspective, enterprise consumers have value in being able to specify a modified timestamp to help _checkpoint_ what deltas they're interested in consuming.
Granted, this context is specific to non-streaming file data sources. I was hopeful to find an equivalent perhaps with Structured Streaming but the closest I found was _latestFirst_ and _maxFileAge_ which each have their respective use cases but does not solve this particular one. The connective tissue between my change here lies in the fact that Structured Streaming also leverages InMemoryFileIndex and actively passes a parameter map to its constructor. I'll provide a PR to complete support there, as well, but separately from this MVP piece.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663915279
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r472579334
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/pathFilters.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.util.{Locale, TimeZone}
+
+import org.apache.hadoop.fs.{FileStatus, GlobFilter}
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.unsafe.types.UTF8String
+
+
+trait PathFilterStrategy extends Serializable {
+ def accept(fileStatus: FileStatus): Boolean
+}
+
+trait StrategyBuilder {
+ def strategy: String
+ def create(parameters: CaseInsensitiveMap[String]): PathFilterStrategy
+}
+
+class PathGlobFilter(filePatten: String) extends PathFilterStrategy {
+
+ private val globFilter = new GlobFilter(filePatten)
+
+ override def accept(fileStatus: FileStatus): Boolean = globFilter.accept(fileStatus.getPath)
+}
+
+object PathGlobFilter extends StrategyBuilder {
+
+ override def strategy: String = "pathglobfilter"
+
+ override def create(parameters: CaseInsensitiveMap[String]): PathFilterStrategy = {
+ new PathGlobFilter(parameters(strategy))
+ }
+}
+
+/**
+ * Provide modifiedAfter and modifiedBefore options when
+ * filtering from a batch-based file data source.
+ *
+ * Example Usages
+ * Load all CSV files modified after date:
+ * {{{
+ * spark.read.format("csv").option("modifiedAfter","2020-06-15T05:00:00").load()
+ * }}}
+ *
+ * Load all CSV files modified before date:
+ * {{{
+ * spark.read.format("csv").option("modifiedBefore","2020-06-15T05:00:00").load()
+ * }}}
+ *
+ * Load all CSV files modified between two dates:
+ * {{{
+ * spark.read.format("csv").option("modifiedAfter","2019-01-15T05:00:00")
+ * .option("modifiedBefore","2020-06-15T05:00:00").load()
+ * }}}
+ */
+abstract class ModifiedDateFilter extends PathFilterStrategy {
+
+ def timeZoneId: String
+
+ protected def localTime(micros: Long): Long = DateTimeUtils.fromUTCTime(micros, timeZoneId)
+}
+
+object ModifiedDateFilter {
+
+ def getTimeZoneId(options: CaseInsensitiveMap[String]): String = {
+ options.getOrElse(
+ DateTimeUtils.TIMEZONE_OPTION.toLowerCase(Locale.ROOT),
+ SQLConf.get.sessionLocalTimeZone)
+ }
+
+ def toThreshold(timeString: String, timeZoneId: String, strategy: String): Long = {
+ val timeZone: TimeZone = DateTimeUtils.getTimeZone(timeZoneId)
+ val ts = UTF8String.fromString(timeString)
+ DateTimeUtils.stringToTimestamp(ts, timeZone.toZoneId).getOrElse {
+ throw new AnalysisException(
+ s"The timestamp provided for the '$strategy' option is invalid. The expected format " +
+ s"is 'YYYY-MM-DDTHH:mm:ss', but the provided timestamp: $timeString")
+ }
+ }
+}
+
+/**
+ * Filter used to determine whether file was modified before the provided timestamp.
+ */
+class ModifiedBeforeFilter(thresholdTime: Long, val timeZoneId: String) extends ModifiedDateFilter {
+
+ override def accept(fileStatus: FileStatus): Boolean =
+ // We standardize on microseconds wherever possible
+ // getModificationTime returns in milliseconds
+ thresholdTime - localTime(DateTimeUtils.millisToMicros(fileStatus.getModificationTime)) > 0
+}
+
+object ModifiedBeforeFilter extends StrategyBuilder {
+ import ModifiedDateFilter._
+
+ override val strategy: String = "modifiedbefore"
+
+ override def create(parameters: CaseInsensitiveMap[String]): PathFilterStrategy = {
+ val timeZoneId = getTimeZoneId(parameters)
+ val thresholdTime = toThreshold(parameters(strategy), timeZoneId, strategy)
+ new ModifiedBeforeFilter(thresholdTime, timeZoneId)
+ }
+}
+
+/**
+ * Filter used to determine whether file was modified after the provided timestamp.
+ */
+class ModifiedAfterFilter(thresholdTime: Long, val timeZoneId: String) extends ModifiedDateFilter {
+
+ override def accept(fileStatus: FileStatus): Boolean =
+ // getModificationTime returns in milliseconds
+ // We standardize on microseconds wherever possible
Review comment:
nit: one more indent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-665558362
**[Test build #126768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126768/testReport)** for PR 28841 at commit [`b925622`](https://github.com/apache/spark/commit/b925622c6574ce03c25a138ed3f43aa6b800b2b5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650055403
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-646447828
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r463960973
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
##########
@@ -561,6 +564,367 @@ class FileBasedDataSourceSuite extends QueryTest
}
}
+ test("SPARK-31962 - when modifiedAfter specified " +
+ "with a past date") {
+ withTempDir { dir =>
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
Review comment:
Could you add tests for multiple file cases? Probably, I think you might be able to use `(new File("/tmp/file.csv")).setLastModified(xxx)` to control timestamp.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-675227409
**[Test build #127524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127524/testReport)** for PR 28841 at commit [`263dd2a`](https://github.com/apache/spark/commit/263dd2a58ee990600aae3c40ea3eb56368a9c48d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r472579411
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/pathFilters.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.util.{Locale, TimeZone}
+
+import org.apache.hadoop.fs.{FileStatus, GlobFilter}
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.unsafe.types.UTF8String
+
+
+trait PathFilterStrategy extends Serializable {
+ def accept(fileStatus: FileStatus): Boolean
+}
+
+trait StrategyBuilder {
+ def strategy: String
+ def create(parameters: CaseInsensitiveMap[String]): PathFilterStrategy
+}
+
+class PathGlobFilter(filePatten: String) extends PathFilterStrategy {
+
+ private val globFilter = new GlobFilter(filePatten)
+
+ override def accept(fileStatus: FileStatus): Boolean = globFilter.accept(fileStatus.getPath)
+}
+
+object PathGlobFilter extends StrategyBuilder {
+
+ override def strategy: String = "pathglobfilter"
+
+ override def create(parameters: CaseInsensitiveMap[String]): PathFilterStrategy = {
+ new PathGlobFilter(parameters(strategy))
+ }
+}
+
+/**
+ * Provide modifiedAfter and modifiedBefore options when
+ * filtering from a batch-based file data source.
+ *
+ * Example Usages
+ * Load all CSV files modified after date:
+ * {{{
+ * spark.read.format("csv").option("modifiedAfter","2020-06-15T05:00:00").load()
+ * }}}
+ *
+ * Load all CSV files modified before date:
+ * {{{
+ * spark.read.format("csv").option("modifiedBefore","2020-06-15T05:00:00").load()
+ * }}}
+ *
+ * Load all CSV files modified between two dates:
+ * {{{
+ * spark.read.format("csv").option("modifiedAfter","2019-01-15T05:00:00")
+ * .option("modifiedBefore","2020-06-15T05:00:00").load()
+ * }}}
+ */
+abstract class ModifiedDateFilter extends PathFilterStrategy {
+
+ def timeZoneId: String
+
+ protected def localTime(micros: Long): Long = DateTimeUtils.fromUTCTime(micros, timeZoneId)
+}
+
+object ModifiedDateFilter {
+
+ def getTimeZoneId(options: CaseInsensitiveMap[String]): String = {
+ options.getOrElse(
+ DateTimeUtils.TIMEZONE_OPTION.toLowerCase(Locale.ROOT),
+ SQLConf.get.sessionLocalTimeZone)
+ }
+
+ def toThreshold(timeString: String, timeZoneId: String, strategy: String): Long = {
+ val timeZone: TimeZone = DateTimeUtils.getTimeZone(timeZoneId)
+ val ts = UTF8String.fromString(timeString)
+ DateTimeUtils.stringToTimestamp(ts, timeZone.toZoneId).getOrElse {
+ throw new AnalysisException(
+ s"The timestamp provided for the '$strategy' option is invalid. The expected format " +
+ s"is 'YYYY-MM-DDTHH:mm:ss', but the provided timestamp: $timeString")
+ }
+ }
+}
+
+/**
+ * Filter used to determine whether file was modified before the provided timestamp.
+ */
+class ModifiedBeforeFilter(thresholdTime: Long, val timeZoneId: String) extends ModifiedDateFilter {
+
+ override def accept(fileStatus: FileStatus): Boolean =
+ // We standardize on microseconds wherever possible
+ // getModificationTime returns in milliseconds
Review comment:
nit: one more indent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r468952791
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PathFilterSuite.scala
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.File
+import java.time.LocalDateTime
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.util.{stringToFile, CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.pathfilters.PathFilterFactory
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+
+class PathFilterSuite
+ extends QueryTest
+ with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ test("SPARK-31962: when modifiedAfter specified with a past date") {
+ withTempDir { dir =>
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ file.setLastModified(DateTimeUtils.currentTimestamp())
+ val df = spark.read
+ .option("modifiedAfter", "2019-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore specified with a future date") {
+ withTempDir { dir =>
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ val df = spark.read
+ .option("modifiedBefore", "2090-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore specified with a past date") {
+ withTempDir { dir =>
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ file.setLastModified(DateTimeUtils.currentTimestamp())
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedBefore", "1984-05-01T01:00:00")
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with a past date, " +
+ "multiple files, one valid") {
+ withTempDir { dir =>
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ val path = new Path(dir.getCanonicalPath)
+ val file1 = new File(dir, "file1.csv")
+ val file2 = new File(dir, "file2.csv")
+ stringToFile(file1, "text")
+ stringToFile(file2, "text")
+ file1.setLastModified(DateTimeUtils.currentTimestamp())
+ file2.setLastModified(0)
+ val df = spark.read
+ .option("modifiedAfter", "2019-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+ }
+ test("SPARK-31962: when modifiedAfter specified with a past date, " +
+ "multiple files, both valid") {
Review comment:
What's a difference between this test and https://github.com/apache/spark/pull/28841/files#diff-9bb59882dc9a25fa8355a712f8ca964eR38 ? I think the two test has the almost same code path, so only this test looks enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673191390
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650900831
> > I wonder though if structured streaming always implied an event source, particularly when streaming from a file source?
>
> Ideally it should be. It's not 100% really for file stream source (as it doesn't store offset and allows reading late files), but it's partially considered as file stream source "sorts" the files by its timestamp, in both directions, forward and backward. Probably you're thinking about "offset" as a file offset, while I mean "offset" to mark where we read so far. In this case, modified time, with some processed files as there could be new files having same modified time (some of files may not be picked up due to max number of files per trigger).
>
> > For example, modifiedDateFilter applies specifically to a point in time when you begin structured streaming on a file data source. You would not have an offset yet in commitedOffsets. The offset use case would imply you are restarting an existing, checkpointed stream or attempting to read from a checkpointed location, right?
>
> Yes, but you'll want to consider the case where end users "changes" modifiedDataFilter before restaring the query from the checkpoint. Every options can be changed during restart, which makes things very complicated. You'll need to consider combinations of `modifiedDataFilter`, `latestFirst`, `maxFilesAge`. And also make sure the behavior is intuitive to the end users. Please refer #28422 for what I proposed and which review comments it got.
>
> > When using an offset with a Kafka data source, some write has occurred by which a written checkpoint exists. With the file data source for files that have not yet been read or written, I'm curious how I would apply offset bounds in this way. I was thinking I would have to be reading from a data source that had used structured streaming with checkpointing in order for the offset to exist (committed).
> >
> > Does this make sense? It seems like once you've written a checkpoint while writing to a stream from the readStream dataframe that's loading files, you would have clear context to apply offset-based semantics.
>
> Yeah, that should be ideally an "offset" instead of file stream source's own metadata, but I'm not sure how far we want to go. Probably I've been thinking too far, but there's a real issue on file stream source metadata growing, and I support the way it also clearly fixes the issue. I'm worried that once we introduce a relevent option without enough consideration, that would bug us via respecting backward compatiblity.
>
> > With_startingOffsetByTimestamp_, you have the ability to indicate start/end offsets per topic such as TopicA or TopicB. If this concept were applied to a file data source with the underlying intent that each file name represented a topic, problems begin to emerge. For example, if there are multiple files, they would have different file names, different file names may imply a new topic.
>
> `underlying intent that each file name represented a topic` - I don't think so (If you thought I proposed it then I didn't mean that). Do you remember a file is an unit of processing on file source? Spark doesn't process the file partially, in other words, Spark never splits the file across micro-batches. If you reflect the file source as Kafka data source, an record in Kafka data source = a file in File data source.
>
> So the timestamp can represent the offset how Spark has been read so far - though there're some issues below:
>
> 1) We allowed to read late files - there's new updated file but modified time being set to "previous time". I don't think it should be, but current File data source allows to do so, so that's arguable topic.
>
> 2) There can be lots of files in same timestamp, and Spark has max files to process in a batch. That means there's a case only partial files get processed. That's why only storing timestamp doesn't work here and we still need to store some processed files as well to mitigate.
>
> 3) That's technically really odd, but considering eventual consistency like S3... Listing operation may not show some of available files. This may be a blocker to use timestamp as an offset, but even with eventual consistency we don't expect it shows after days. (Do I understand correctly on expectation for eventual consistency on S3?) That said, (ts for (timestamp - some margin) + processed files within margin) could be used for timestamp instead. We can leverage purge timestamp in SeenFileMap, though I think 7 days are too long. And latestFirst should be replaced with something else, because it just simply defeats maxFilesAge and unable to apply offset-like approach.
>
> Does it make sense?
We were writing responses at the same time. Just saw this and will read through
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-675305213
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-680090397
@gengliangwang @dongjoon-hyun @cloud-fan Anyone could review this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman edited a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman edited a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-665448306
@HeartSaVioR
To reiterate what kicked off the last series of commits here, there was a conflict that needed to be resolved with readwriter.py where a new parameter was added. I reverted my accidental commit and resolved the file in question. Everything should be back in order,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721810364
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672156463
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r463954939
##########
File path: docs/sql-data-sources-generic-options.md
##########
@@ -119,3 +119,37 @@ To load all files recursively, you can use:
{% include_example recursive_file_lookup r/RSparkSQLExample.R %}
</div>
</div>
+
+### Modification Time Path Filters
+`modifiedBefore` and `modifiedAfter` are options that can be
+applied together or separately in order to achieve greater
+granularity over which files may load during a Spark batch query. <br/>
+<br/>When the `timeZone` option is present, modified timestamps will be
+interpreted according to the specified zone. When a timezone option
+is not provided, modified timestamps will be interpreted according
+to the default zone specified within the Spark configuration. Without
+any timezone configuration, modified timestamps are interpreted as UTC.
+
+`modifiedBefore` will only allow files having last modified
+timestamps occurring before the specified time to load. For example,
+when`modifiedBefore`has the timestamp `2020-06-01T12:00:00` applied,
+ all files modified after that time will not be considered when loading from a file data source.<br/><br/>
+`modifiedAfter` only allows files having last modified timestamps
+occurring after the specified timestamp. For example, when`modifiedAfter`
+has the timestamp `2020-06-01T12:00:00` applied, only files modified after
+ this time will be eligible when loading from a file data source.
+ <br/><br/>
+When both `modifiedBefore` and `modifiedAfter` are specified together, files having
+last modified timestamps within the resulting time range are the only files
+allowed to load.
+<br/><br/>
+Both options expect a timestamp in the following format: `YYYY-MM-DDTHH:mm:ss`.<br/>
+
+<b>Example</b> - <i>Load files between 6/1 and 7/1 time range:</i><br/>
+`spark`<br/>
+`.read` <br/>
+` .format("csv")`<br/>
+` .option('modifiedBefore','2020-07-01T08:33:05`)<br/>
+` .option('modifiedAfter','2020-06-01T08:33:05`)<br/>
+` .option('timeZone','PST`)<br/>
Review comment:
Could you follow the format by referring the other `.md` docs? You can generate a html doc and check it via `SKIP_API=1 jekyll build`.
<img width="1030" alt="Screen Shot 2020-08-01 at 20 49 50" src="https://user-images.githubusercontent.com/692303/89101208-b0577480-d438-11ea-8021-d7a9555d3bcf.png">
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/pathfilters/ModifiedAfterFilter.scala
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.pathfilters
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs._
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateTimeUtils}
+
+/**
+ * [SPARK-31962]
+ * Filter used to determine whether file was modified
+ * after the provided timestamp.
+ *
+ * @param sparkSession SparkSession
+ * @param hadoopConf Hadoop Configuration object
+ * @param options Map containing options
+ */
+case class ModifiedAfterFilter(sparkSession: SparkSession,
+ hadoopConf: Configuration,
+ options: CaseInsensitiveMap[String])
+ extends ModifiedDateFilter(sparkSession, hadoopConf, options)
Review comment:
Could you check the style guide again?
https://github.com/databricks/scala-style-guide#spacing-and-indentation
```
case class ModifiedAfterFilter(
sparkSession: SparkSession,
hadoopConf: Configuration,
options: CaseInsensitiveMap[String])
extends ModifiedDateFilter(sparkSession, hadoopConf, options) with FileIndexFilter {
```
##########
File path: python/pyspark/sql/readwriter.py
##########
@@ -114,6 +114,14 @@ def option(self, key, value):
* ``pathGlobFilter``: an optional glob pattern to only include files with paths matching
the pattern. The syntax follows org.apache.hadoop.fs.GlobFilter.
It does not change the behavior of partition discovery.
+ * ``modifiedBefore``: an optional timestamp to only include files with
+ modification times occurring before the specified time. The provided timestamp
+ must be in the following format: YYYY-MM-DDTHH:mm:ss
Review comment:
Could you use a single space in `format: YYYY-`?
##########
File path: docs/sql-data-sources-generic-options.md
##########
@@ -119,3 +119,37 @@ To load all files recursively, you can use:
{% include_example recursive_file_lookup r/RSparkSQLExample.R %}
</div>
</div>
+
+### Modification Time Path Filters
+`modifiedBefore` and `modifiedAfter` are options that can be
+applied together or separately in order to achieve greater
+granularity over which files may load during a Spark batch query. <br/>
+<br/>When the `timeZone` option is present, modified timestamps will be
+interpreted according to the specified zone. When a timezone option
+is not provided, modified timestamps will be interpreted according
+to the default zone specified within the Spark configuration. Without
+any timezone configuration, modified timestamps are interpreted as UTC.
+
+`modifiedBefore` will only allow files having last modified
+timestamps occurring before the specified time to load. For example,
+when`modifiedBefore`has the timestamp `2020-06-01T12:00:00` applied,
+ all files modified after that time will not be considered when loading from a file data source.<br/><br/>
+`modifiedAfter` only allows files having last modified timestamps
+occurring after the specified timestamp. For example, when`modifiedAfter`
+has the timestamp `2020-06-01T12:00:00` applied, only files modified after
+ this time will be eligible when loading from a file data source.
+ <br/><br/>
+When both `modifiedBefore` and `modifiedAfter` are specified together, files having
+last modified timestamps within the resulting time range are the only files
+allowed to load.
+<br/><br/>
+Both options expect a timestamp in the following format: `YYYY-MM-DDTHH:mm:ss`.<br/>
+
+<b>Example</b> - <i>Load files between 6/1 and 7/1 time range:</i><br/>
+`spark`<br/>
+`.read` <br/>
+` .format("csv")`<br/>
+` .option('modifiedBefore','2020-07-01T08:33:05`)<br/>
+` .option('modifiedAfter','2020-06-01T08:33:05`)<br/>
+` .option('timeZone','PST`)<br/>
Review comment:
Also, please add `{% include_example load_with_modified_time_filter... `
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -467,6 +467,14 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* <li>`pathGlobFilter`: an optional glob pattern to only include files with paths matching
* the pattern. The syntax follows <code>org.apache.hadoop.fs.GlobFilter</code>.
* It does not change the behavior of partition discovery.</li>
+ * <li>`modifiedBefore`: an optional timestamp to only include files with
+ * modification times occurring before the specified time. The provided timestamp
+ * must be in the following form: <code>YYYY-MM-DDTHH:mm:ss</code> Example:
Review comment:
ditto: Could you use a single space in `form: <code>`?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
##########
@@ -561,6 +564,367 @@ class FileBasedDataSourceSuite extends QueryTest
}
}
+ test("SPARK-31962 - when modifiedAfter specified " +
Review comment:
nit: `SPARK-31962 -` -> `SPARK-31962: `
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/pathfilters/ModifiedAfterFilter.scala
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.pathfilters
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs._
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateTimeUtils}
+
+/**
+ * [SPARK-31962]
Review comment:
Could you remove the jira ID? I think we don't need it here.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/pathfilters/FileIndexFilter.scala
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.pathfilters
+
+import org.apache.hadoop.fs.{FileStatus, PathFilter}
+
+
+trait FileIndexFilter extends PathFilter with Serializable {
+ def accept(fileStatus: FileStatus): Boolean
+ def strategy(): String
Review comment:
We need `def strategy(): String`? Looks we can use a class name instead.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/pathfilters/FileIndexFilter.scala
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.pathfilters
+
+import org.apache.hadoop.fs.{FileStatus, PathFilter}
+
+
+trait FileIndexFilter extends PathFilter with Serializable {
Review comment:
Could you make a file `org/apache/spark/sql/execution/datasources/pathFilters.scala`, then move all the related classes into the file? I think each new file has less code.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -467,6 +467,14 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* <li>`pathGlobFilter`: an optional glob pattern to only include files with paths matching
* the pattern. The syntax follows <code>org.apache.hadoop.fs.GlobFilter</code>.
* It does not change the behavior of partition discovery.</li>
+ * <li>`modifiedBefore`: an optional timestamp to only include files with
+ * modification times occurring before the specified time. The provided timestamp
+ * must be in the following form: <code>YYYY-MM-DDTHH:mm:ss</code> Example:
+ * <code>2020-06-01T13:00:00</code>
+ * <li>`modifiedAfter`: an optional timestamp to only include files with
+ * modification times occurring after the specified time. The provided timestamp
Review comment:
ditto: `time. The`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-664116633
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126610/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693370587
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721740649
**[Test build #130602 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130602/testReport)** for PR 28841 at commit [`40c99dc`](https://github.com/apache/spark/commit/40c99dccb659b56f531144dcbc57598e8914e4ba).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-664161094
**[Test build #126611 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126611/testReport)** for PR 28841 at commit [`b7952a6`](https://github.com/apache/spark/commit/b7952a6af9694354972d8cb4cfc0548e6cecc1f8).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663839623
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721762845
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35203/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672641802
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721756841
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-720292747
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650239439
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r449241653
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##########
@@ -83,7 +85,7 @@ abstract class PartitioningAwareFileIndex(
val files: Seq[FileStatus] = leafDirToChildrenFiles.get(path) match {
case Some(existingDir) =>
// Directory has children files in it, return them
- existingDir.filter(f => matchGlobPattern(f) && isNonEmptyFile(f))
+ existingDir.filter(f => matchPathPattern(f) && isNonEmptyFile(f))
Review comment:
> `f` here is `FileStatus`. It looks inefficient to get the `FileStatus` again in https://github.com/apache/spark/pull/28841/files#diff-3c816e65d0bbdf16d46c7ea4e0b8cbaeR55
I agree. I considered this in one of my earlier commits actually. There are two places where we handle options in the *FileIndex classes, here in _PartitioningAwareFileIndex_ and _InMemoryFileIndex_. In both places, we derive from the _PathFilter_ class which exposes an overridable method called _accept_ which takes _Path_ as its only parameter.
I could break with the abstraction and overload the method so that it would take a FileStatus directly. I'll look into this unless there are any objections.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693589479
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650112787
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650037599
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124533/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-679882932
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-675301748
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693227460
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672641442
**[Test build #127369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127369/testReport)** for PR 28841 at commit [`3ded6d0`](https://github.com/apache/spark/commit/3ded6d0f582468913654d48c9ff4cb3658526697).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-687671184
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721756849
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721819550
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35207/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721797454
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35206/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-687593777
**[Test build #128319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128319/testReport)** for PR 28841 at commit [`d57bf08`](https://github.com/apache/spark/commit/d57bf088bd581bd5cd1c1ea81f44581a1f20a6ed).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721732961
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130600/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-680017499
**[Test build #127883 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127883/testReport)** for PR 28841 at commit [`263dd2a`](https://github.com/apache/spark/commit/263dd2a58ee990600aae3c40ea3eb56368a9c48d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-671990635
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-652857478
Your math is correct and I agree it helps. The thing is, how much it would help?
When we talk about streaming we are probably talking about the query which runs months. The lower bound is static and the query will get boosted on the earlier batch to filter out older files, but once the query catches up, situation would be similar.
I agree the overall complication is not a goal for this PR - just wanted to allow me think less when I go through such complication. One more option, one more complication.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663921502
**[Test build #126559 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126559/testReport)** for PR 28841 at commit [`0e97f5e`](https://github.com/apache/spark/commit/0e97f5edded8b80ec94d48a3ff0d595bca373c3e).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r450584173
##########
File path: docs/sql-data-sources-generic-options.md
##########
@@ -119,3 +119,31 @@ To load all files recursively, you can use:
{% include_example recursive_file_lookup r/RSparkSQLExample.R %}
</div>
</div>
+
+### Modification Date Filter
+
+`modifiedDateFilter` is an option used to only load files after a specified modification
Review comment:
If you agree with this approach, I can be sure any reference to date in above, docs, etc instead use time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721799049
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35205/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-722438008
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721761135
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650726701
> Please take a look at how Kafka data source options apply with both batch and streaming query. The semantic of the option should be applied differently.
>
> http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#creating-a-kafka-source-for-batch-queries
>
> `startingOffsetsByTimestamp`, `startingOffsets`, `endingOffsetsByTimestamp`, `endingOffsets`
>
> If we are not fully sure about how to do it, let's only apply the option to batch query, and file an issue to address for the streaming query.
>
> Btw, that said, I prefer to have lower bound + upper bound instead of only lower bound, as commented earlier on reviewing.
@HeartSaVioR
Hmm, I'm wondering if this isn't a different feature. The goal of this feature is to begin reading from a file data source with files that have a particular modified date. It's key value is really with having the ability to _start_ at a particular _physical_ location.
With a Kafka data source, you're exclusively dealing with an event stream where event sourcing patterns leveraging offsets are at play. I wonder though if structured streaming always implied an event source, particularly when streaming from a file source?
For example, _modifiedDateFilter_ applies specifically to a point in time when you begin structured streaming on a file data source. You would not have an offset yet in _availableOffsets_. The offset use case would imply you are restarting an existing, checkpointed stream.
When using an offset with a Kafka data source, some write has occurred by which a written checkpoint exists. With the file data source for files that have not yet been read or written, I'm curious how I would apply offset bounds in this way. I was thinking I would have to be reading from a data source that had used structured streaming with checkpointing in order for the offset to exist (committed).
Does this make sense? It seems like once you've written a checkpoint while writing to a stream from the readStream dataframe that's loading files, you would have clear context to apply offset-based semantics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-665558362
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673864612
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127431/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-647012574
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663846750
**[Test build #126543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126543/testReport)** for PR 28841 at commit [`17de795`](https://github.com/apache/spark/commit/17de795b8ff1726c2a938a97fc84f1f13f02f3a5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663850354
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663913337
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-680150141
**[Test build #127889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127889/testReport)** for PR 28841 at commit [`3a99d90`](https://github.com/apache/spark/commit/3a99d905989083d9828fcab9d3ddf37cc0caa72e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-652894802
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-680018752
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-687671059
**[Test build #128321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128321/testReport)** for PR 28841 at commit [`1c8384c`](https://github.com/apache/spark/commit/1c8384cebe05fbc1ebb3c86660e57508b031c6f9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-664703222
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663850911
**[Test build #126546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126546/testReport)** for PR 28841 at commit [`e62568a`](https://github.com/apache/spark/commit/e62568ab07ffcf0dfa760bb09e44cee27e2bddd8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672661230
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #28841:
URL: https://github.com/apache/spark/pull/28841
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650055403
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-695160233
> These timestamps should be inclusive or exclusive? e.g., `(modifiedAfter, modifiedBefore)`, `[modifiedAfter, modifiedBefore]`... Looks like the current code follows `(modifiedAfter, modifiedBefore]`? Anyway, I think we need to test all the boundary cases.
@maropu's request to explore these type of conditions as test cases led to finding a corner case that was missed. Four tests were added to provide protection around the bounded contexts of our timeZone parameter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721761148
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130606/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721810364
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663846951
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126543/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-695158320
> > The need arises since we only have one filtering strategy where pathGlobFilter is used in PartitioningAwareFileIndex and handling cleanly if we now have more than one approach to path filtering so the implementation I'm suggesting would be nearly identical to above except moving it to where pathGlobFilter is implemented.
>
> @cchighman the option should be effective for both `MetadataLogFileIndex` and `InMemoryFileIndex`. That's why I suggest implementing it in `PartitioningAwareFileIndex`. I think we can create a method to determine which files are matched according to the parameters.
Further feedback from @gengliangwang provided clear direction for fine-tuning this implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-665559113
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r463961062
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
##########
@@ -561,6 +564,367 @@ class FileBasedDataSourceSuite extends QueryTest
}
}
+ test("SPARK-31962 - when modifiedAfter specified " +
+ "with a past date") {
+ withTempDir { dir =>
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ file.setLastModified(DateTimeUtils.currentTimestamp())
+ val df = spark.read
+ .option("modifiedAfter", "2019-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+ }
+
+ test("SPARK-31962 - when modifiedBefore specified " +
+ "with a future date") {
Review comment:
ditto: you don't need the line break.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672662992
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127361/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-722329265
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672557773
**[Test build #127358 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127358/testReport)** for PR 28841 at commit [`cf9d2fa`](https://github.com/apache/spark/commit/cf9d2fa70c88ece8f377c12967c84602f56cc9b4).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693368220
**[Test build #128752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128752/testReport)** for PR 28841 at commit [`1c8384c`](https://github.com/apache/spark/commit/1c8384cebe05fbc1ebb3c86660e57508b031c6f9).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ModifiedBeforeFilter(thresholdTime: Long, val timeZoneId: String)`
* `class ModifiedAfterFilter(thresholdTime: Long, val timeZoneId: String)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-722330354
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-646395368
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-647809034
> > I haven't looked into the detail yet, but Hadoop FS and a PathFilter API, does Spark file source support the path filter option?
>
> Hadoop's PathFilter only gives the user Path without metadata. But this is an interesting point. The user needs a file metadata filter essentially. For example, some user may ask to filter files using the file size. But supporting a filter in multiple languages is hard.
The metadata filter sounds like a great extension.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r468634664
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
##########
@@ -561,6 +564,367 @@ class FileBasedDataSourceSuite extends QueryTest
}
}
+ test("SPARK-31962 - when modifiedAfter specified " +
+ "with a past date") {
+ withTempDir { dir =>
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ file.setLastModified(DateTimeUtils.currentTimestamp())
+ val df = spark.read
+ .option("modifiedAfter", "2019-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+ }
+
+ test("SPARK-31962 - when modifiedBefore specified " +
+ "with a future date") {
Review comment:
Completed
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
##########
@@ -561,6 +564,367 @@ class FileBasedDataSourceSuite extends QueryTest
}
}
+ test("SPARK-31962 - when modifiedAfter specified " +
+ "with a past date") {
Review comment:
Completed
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
##########
@@ -561,6 +564,367 @@ class FileBasedDataSourceSuite extends QueryTest
}
}
+ test("SPARK-31962 - when modifiedAfter specified " +
+ "with a past date") {
+ withTempDir { dir =>
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
Review comment:
Six more tests were added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-664176747
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672156481
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127331/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-679990468
**[Test build #127889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127889/testReport)** for PR 28841 at commit [`3a99d90`](https://github.com/apache/spark/commit/3a99d905989083d9828fcab9d3ddf37cc0caa72e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-681765576
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672562400
**[Test build #127362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127362/testReport)** for PR 28841 at commit [`50ebdfc`](https://github.com/apache/spark/commit/50ebdfc5f2611f40aec2adaf4c6aa1f224472349).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650082905
**[Test build #124541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124541/testReport)** for PR 28841 at commit [`b4e0428`](https://github.com/apache/spark/commit/b4e042851506880a6c8515d99bc344a26d937283).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721732932
**[Test build #130600 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130600/testReport)** for PR 28841 at commit [`4480ca7`](https://github.com/apache/spark/commit/4480ca7bfbf8a5bce19ab9c3c364d5866e3ea135).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672566232
@maropu
Everything should be updated as requested. I appreciate your thorough review and believe we arrived at a better result in the process.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-722330363
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35258/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-690007273
@maropu @HeartSaVioR @gengliangwang @cloud-fan @Dooyoung-Hwang
Gentle ping. Thank you for your time!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-687691634
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-722290335
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673864606
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663941538
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-653338249
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650051641
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r450583791
##########
File path: docs/sql-data-sources-generic-options.md
##########
@@ -119,3 +119,31 @@ To load all files recursively, you can use:
{% include_example recursive_file_lookup r/RSparkSQLExample.R %}
</div>
</div>
+
+### Modification Date Filter
+
+`modifiedDateFilter` is an option used to only load files after a specified modification
Review comment:
@gengliangwang
This is out of date here after recent comments from @HeartSaVioR. I updated the PR commit above and title based on his feedback. I'm currently working on these additions with the following:
**Example Usages**
_Load all CSV files modified after date:_
`spark.read.format("csv").option("modifiedAfter","2020-06-15T05:00:00").load()`
_Load all CSV files modified before date:_
`spark.read.format("csv").option("modifiedBefore","2020-06-15T05:00:00").load()`
_Load all CSV files modified between two dates:_
`spark.read.format("csv").option("modifiedAfter","2019-01-15T05:00:00").option("modifiedBefore","2020-06-15T05:00:00").load()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-647002537
**[Test build #124324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124324/testReport)** for PR 28841 at commit [`2126c85`](https://github.com/apache/spark/commit/2126c857b4c34d5983afe88def25c90bc4f94df7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r443087420
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
##########
@@ -356,26 +380,35 @@ object InMemoryFileIndex extends Logging {
bulkListLeafFiles(
dirs.map(_.getPath),
hadoopConf,
- filter,
+ filters,
session,
- areRootPaths = false
+ areRootPaths = false,
+ parameters = parameters
).flatMap(_._2)
case _ =>
dirs.flatMap { dir =>
listLeafFiles(
dir.getPath,
hadoopConf,
- filter,
+ filters,
sessionOpt,
ignoreMissingFiles = ignoreMissingFiles,
ignoreLocality = ignoreLocality,
- isRootPath = false)
+ isRootPath = false,
+ parameters = parameters)
}
}
+ val fileFilters = (filters ++ FileModifiedDateOption.accept(
+ parameters,
+ sessionOpt.get,
Review comment:
You can choose some tests in the following report in order to verify your fix. Please push your commit after passing those UT locally.
- https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124247/testReport/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721756841
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673868046
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-647012367
**[Test build #124326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124326/testReport)** for PR 28841 at commit [`918af78`](https://github.com/apache/spark/commit/918af78219225dd130a36d539212659fc9c9df8c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650519071
**[Test build #124564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124564/testReport)** for PR 28841 at commit [`7ee667d`](https://github.com/apache/spark/commit/7ee667d7a253e4ab1f70826b5a36177865d479c5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672561005
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-647838170
> @gengliangwang @dongjoon-hyun
>
> Are there any strong feelings against making `pathGlobfilter` a derived class strategy, along with the disjointed implementation in `CommandUtils` for `PathIgnoreNonData`, as been implemented here in `InMemoryFileIndex` except that the implementation approach and strategy would actually be according to `pathGlobFilter` in `PartitioningAwareFileIndex`?
>
> In effect, the implementation in my change above in `listLeafFiles` was extending what appears to be a duplication of the initial strategy used here for `pathGlobFilter`. In `PartitioningAwareFileIndex`, the `allFiles` method uses the same filtering methodology.
>
> So, the difference would be the below method signature taking the same strategy approach on the abstraction to determine which filters to apply:
>
> ` protected def matchGlobPattern(file: FileStatus): Boolean = { pathGlobFilter.forall(_.accept(file.getPath)) }`
>
> _changes to_
>
> ` protected def matchPathFilter(file: FileStatus): Boolean = { pathFilters.forall(_.accept(file.getPath)) }`
>
> This would mean there is little implementation changes in the current code and it can be easily extended. `GlobFilter`, being a derived class from `PathFilter`, makes this a rather perfect fit since each case is implementing the `accept` interface. We would also be consolidating these two different filtering approaches in the same place, reducing code, as to eliminate any sort of confusion in the future.
>
> I can't see any discernable benefit as to why `InMemoryFileIndex` needs a separate implementation in `listLeafFiles` for path filtering that isn't already handled when filtering during `allfiles` where `pathGlobFilter` is implemented.
> @gengliangwang @dongjoon-hyun
>
> Are there any strong feelings against making `pathGlobfilter` a derived class strategy, along with the disjointed implementation in `CommandUtils` for `PathIgnoreNonData`, as been implemented here in `InMemoryFileIndex` except that the implementation approach and strategy would actually be according to `pathGlobFilter` in `PartitioningAwareFileIndex`?
>
> In effect, the implementation in my change above in `listLeafFiles` was extending what appears to be a duplication of the initial strategy used here for `pathGlobFilter`. In `PartitioningAwareFileIndex`, the `allFiles` method uses the same filtering methodology.
>
> So, the difference would be the below method signature taking the same strategy approach on the abstraction to determine which filters to apply:
>
> ` protected def matchGlobPattern(file: FileStatus): Boolean = { pathGlobFilter.forall(_.accept(file.getPath)) }`
>
> _changes to_
>
> ` protected def matchPathFilter(file: FileStatus): Boolean = { pathFilters.forall(_.accept(file.getPath)) }`
>
> This would mean there is little implementation changes in the current code and it can be easily extended. `GlobFilter`, being a derived class from `PathFilter`, makes this a rather perfect fit since each case is implementing the `accept` interface. We would also be consolidating these two different filtering approaches in the same place, reducing code, as to eliminate any sort of confusion in the future.
>
> I can't see any discernable benefit as to why `InMemoryFileIndex` needs a separate implementation in `listLeafFiles` for path filtering that isn't already handled when filtering during `allfiles` where `pathGlobFilter` is implemented.
The need arises since we only have one filtering strategy where `pathGlobFilter` is used in `PartitioningAwareFileIndex` and handling cleanly if we now have more than one approach to path filtering.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650087599
**[Test build #124542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124542/testReport)** for PR 28841 at commit [`4a80fdc`](https://github.com/apache/spark/commit/4a80fdc97565f5d33e64debc9642021783663d5f).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman closed pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman closed pull request #28841:
URL: https://github.com/apache/spark/pull/28841
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663848281
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r475347166
##########
File path: docs/sql-data-sources-generic-options.md
##########
@@ -119,3 +119,48 @@ To load all files recursively, you can use:
{% include_example recursive_file_lookup r/RSparkSQLExample.R %}
</div>
</div>
+
+### Modification Time Path Filters
+`modifiedBefore` and `modifiedAfter` are options that can be
+applied together or separately in order to achieve greater
+granularity over which files may load during a Spark batch query.
+
+When the `timeZone` option is present, modified timestamps will be
+interpreted according to the specified zone. When a timezone option
+is not provided, modified timestamps will be interpreted according
+to the default zone specified within the Spark configuration. Without
+any timezone configuration, modified timestamps are interpreted as UTC.
+
+`modifiedBefore` will only allow files having last modified
+timestamps occurring before the specified time to load. For example,
+when`modifiedBefore` has the timestamp `2020-06-01T12:00:00` applied,
+all files modified after that time will not be considered when loading
+from a file data source.
+
+`modifiedAfter` only allows files having last modified timestamps
+occurring after the specified timestamp. For example, when`modifiedAfter`
+has the timestamp `2020-06-01T12:00:00` applied, only files modified after
+this time will be eligible when loading from a file data source. When both
+`modifiedBefore` and `modifiedAfter` are specified together, files having
+last modified timestamps within the resulting time range are the only files
+allowed to load.
Review comment:
Will update
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PathFilterSuite.scala
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.File
+import java.time.{LocalDateTime, ZoneOffset}
+import java.time.format.DateTimeFormatter
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.util.{stringToFile, CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.test.SharedSparkSession
+
+class PathFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ test("SPARK-31962: when modifiedAfter specified with a past date") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ file.setLastModified(DateTimeUtils.currentTimestamp())
+ val df = spark.read
+ .option("modifiedAfter", "2019-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore specified with a future date") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ val df = spark.read
+ .option("modifiedBefore", "2090-05-10T01:11:00")
+ .format("csv")
+ .load(path.toString)
+ assert(df.count() == 1)
+ }
+ }
+
+ test("SPARK-31962: when modifiedBefore specified with a past date") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file = new File(dir, "file1.csv")
+ stringToFile(file, "text")
+ file.setLastModified(DateTimeUtils.currentTimestamp())
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedBefore", "1984-05-01T01:00:00")
+ .format("csv")
+ .load(path.toString)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with a past date, multiple files, one valid") {
+ withTempDir { dir =>
+ val path = new Path(dir.getCanonicalPath)
+ val file1 = new File(dir, "file1.csv")
+ val file2 = new File(dir, "file2.csv")
+ stringToFile(file1, "text")
+ stringToFile(file2, "text")
+ file1.setLastModified(DateTimeUtils.currentTimestamp())
Review comment:
Will Update
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r468972753
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/pathFilters.scala
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.util.{Locale, TimeZone}
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, GlobFilter, Path, PathFilter}
+
+import org.apache.spark.sql.{AnalysisException, SparkSession}
+import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.execution.datasources.pathfilters.PathFilterStrategy
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.unsafe.types.UTF8String
+
+
+/**
+ * SPARK-31962: Provide modifiedAfter and modifiedBefore options when
+ * filtering from a batch-based file data source.
+ *
+ * Example Usages
+ * Load all CSV files modified after date:
+ * spark.read.format("csv").option("modifiedAfter","2020-06-15T05:00:00").load()
+ *
+ * Load all CSV files modified before date:
+ * spark.read.format("csv").option("modifiedBefore","2020-06-15T05:00:00").load()
+ *
+ * Load all CSV files modified between two dates:
+ * spark.read.format("csv").option("modifiedAfter","2019-01-15T05:00:00")
+ * .option("modifiedBefore","2020-06-15T05:00:00").load()
+ *
+@param sparkSession SparkSession
+@param hadoopConf Hadoop Configuration object
+@param options Map containing options
+ */
+abstract class ModifiedDateFilter(sparkSession: SparkSession,
+ hadoopConf: Configuration,
+ options: CaseInsensitiveMap[String])
+ extends PathFilterStrategy(sparkSession, hadoopConf, options) {
+ lazy val timeZoneId: String = options.getOrElse(
Review comment:
Thank you..im going to adjust my IDE settings to correctly reflect these styles.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663913215
**[Test build #126556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126556/testReport)** for PR 28841 at commit [`247b63b`](https://github.com/apache/spark/commit/247b63bced263d668548a74e554cefd9e7f7e581).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673868049
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127432/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663850438
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663840220
**[Test build #126542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126542/testReport)** for PR 28841 at commit [`ac5227d`](https://github.com/apache/spark/commit/ac5227d56ecb09a0892f69af66548e2c8b383512).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650101342
**[Test build #124545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124545/testReport)** for PR 28841 at commit [`d2f201b`](https://github.com/apache/spark/commit/d2f201b803459d6608edc2e7ad7dacc1f8fa82e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-681928591
**[Test build #127950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127950/testReport)** for PR 28841 at commit [`980f103`](https://github.com/apache/spark/commit/980f103a4692e0e8f840e0aafc3f425fd369193c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r449350240
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##########
@@ -83,7 +85,7 @@ abstract class PartitioningAwareFileIndex(
val files: Seq[FileStatus] = leafDirToChildrenFiles.get(path) match {
case Some(existingDir) =>
// Directory has children files in it, return them
- existingDir.filter(f => matchGlobPattern(f) && isNonEmptyFile(f))
+ existingDir.filter(f => matchPathPattern(f) && isNonEmptyFile(f))
Review comment:
@cloud-fan This has been resolved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-683336845
@maropu @HeartSaVioR @cloud-fan @gengliangwang @dongjoon-hyun
Gentle ping to confirm whether all looks good. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r468988356
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PathFilterSuite.scala
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.File
+import java.time.LocalDateTime
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.util.{stringToFile, CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.pathfilters.PathFilterFactory
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+
+class PathFilterSuite
+ extends QueryTest
+ with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
Review comment:
You are correct. This has been removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-645813858
@cchighman I like the idea of such filtering.
How about we follow the approach of the option `pathGlobFilter`: https://github.com/apache/spark/pull/24518/files#diff-e458973c0425211675054a6e6d742957R59
The modification time is also available in `FileStatus`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650036587
**[Test build #124533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124533/testReport)** for PR 28841 at commit [`c9ad703`](https://github.com/apache/spark/commit/c9ad703d0c760d840fe6742d492bde50c8e80fda).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r450576299
##########
File path: examples/src/main/python/sql/datasource.py
##########
@@ -69,6 +69,25 @@ def generic_file_source_options_example(spark):
# +-------------+
# $example off:load_with_path_glob_filter$
+ # $example on:load_with_modified_date_filter$
+ # Assume file in provided path were last modified today.
+ df = spark.read.load("examples/src/main/resources/dir1",
+ format="parquet", modifiedDateFilter="2020-06-01T08:30:00")
Review comment:
Great question by the way..thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-644796120
@bart-samwel To your point, I wonder if "_fromModifiedDate_" would be more appropriate?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-695158061
> `f` here is `FileStatus`. It looks inefficient to get the `FileStatus` again in https://github.com/apache/spark/pull/28841/files#diff-3c816e65d0bbdf16d46c7ea4e0b8cbaeR55
Feedback from @cloud-fan prompted a closer consideration surrounding optimization. The code in question was optimized and resolved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663840222
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721748648
**[Test build #130604 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130604/testReport)** for PR 28841 at commit [`1d54fa5`](https://github.com/apache/spark/commit/1d54fa55c08b12919b1e94f31cf0154ded5133f2).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-720900095
Sigh. Now it has conflicts so I can't go with giving +1 and merging.
@cchighman Could you please fix the conflicts? It would be totally OK if you'd like to let me take this over - just wanted to give a full credit of yours and accelerate the process of merging. If you can deal with fixing conflicts and hopefully also address my minor comments I'll give +1 if tests pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721739507
**[Test build #130602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130602/testReport)** for PR 28841 at commit [`40c99dc`](https://github.com/apache/spark/commit/40c99dccb659b56f531144dcbc57598e8914e4ba).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r494288466
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PathFilterSuite.scala
##########
@@ -0,0 +1,593 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.File
+import java.time.{LocalDateTime, ZoneOffset}
+import java.time.format.DateTimeFormatter
+
+import scala.util.Random
+
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.util.{stringToFile, DateTimeUtils}
+import org.apache.spark.sql.test.SharedSparkSession
+
+class PathFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ def createSingleFile(dir: File): File = {
+ val file = new File(dir, "temp" + Random.nextInt(1000) + ".csv")
+ stringToFile(file, "text")
+ }
+
+ def setFileTime(time: LocalDateTime, file: File): Boolean = {
+ val sameTime = time.toEpochSecond(ZoneOffset.UTC)
+ file.setLastModified(sameTime * 1000)
+ }
+
+ def setPlusFileTime(time: LocalDateTime, file: File, interval: Long): Boolean = {
+ val sameTime = time.plusDays(interval).toEpochSecond(ZoneOffset.UTC)
+ file.setLastModified(sameTime * 1000)
+ }
+
+ def setMinusFileTime(time: LocalDateTime, file: File, interval: Long): Boolean = {
+ val sameTime = time.minusDays(interval).toEpochSecond(ZoneOffset.UTC)
+ file.setLastModified(sameTime * 1000)
+ }
+
+ def formatTime(time: LocalDateTime): String = {
+ time.format(DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss"))
+ }
+
+ test(
+ "SPARK-31962: when modifiedBefore specified" +
+ " and sharing same timestamp with file last modified time.") {
+ withTempDir { dir =>
+ val file = createSingleFile(dir)
+ val time = LocalDateTime.now(ZoneOffset.UTC)
+ setFileTime(time, file)
+ val formattedTime = formatTime(time)
+
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedBefore", formattedTime)
+ .option("timeZone", "UTC")
+ .format("csv")
+ .load(dir.getCanonicalPath)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test(
+ "SPARK-31962: when modifiedAfter specified" +
+ " and sharing same timestamp with file last modified time.") {
+ withTempDir { dir =>
+ val file = createSingleFile(dir)
+ val time = LocalDateTime.now()
+ setFileTime(time, file)
+ val formattedTime = formatTime(time)
+
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", formattedTime)
+ .option("timeZone", "UTC")
+ .format("csv")
+ .load(dir.getCanonicalPath)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test(
+ "SPARK-31962: when modifiedBefore and modifiedAfter option" +
+ " share same timestamp with file last modified time.") {
+ withTempDir { dir =>
+ val file = createSingleFile(dir)
+ val time = LocalDateTime.now()
+ setFileTime(time, file)
+ val formattedTime = formatTime(time)
+
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", formattedTime)
+ .option("modifiedBefore", formattedTime)
+ .format("csv")
+ .load(dir.getCanonicalPath)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test(
+ "SPARK-31962: when modifiedBefore and modifiedAfter option" +
+ " share same timestamp with earlier file last modified time.") {
+ withTempDir { dir =>
+ val file = createSingleFile(dir)
+ val time = LocalDateTime.now()
+ setMinusFileTime(time, file, 3)
+
+ val formattedTime = formatTime(time)
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", formattedTime)
+ .option("modifiedBefore", formattedTime)
+ .format("csv")
+ .load(dir.getCanonicalPath)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test(
+ "SPARK-31962: when modifiedBefore and modifiedAfter option" +
+ " share same timestamp with later file last modified time.") {
+ withTempDir { dir =>
+ createSingleFile(dir)
+ val time = LocalDateTime.now()
+ val formattedTime = formatTime(time)
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", formattedTime)
+ .option("modifiedBefore", formattedTime)
+ .format("csv")
+ .load(dir.getCanonicalPath)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with a past date") {
+ withTempDir { dir =>
+ val file = createSingleFile(dir)
+ file.setLastModified(DateTimeUtils.currentTimestamp())
Review comment:
Shall we do the similar with below test instead of using specific date/format?
Moreover, it would be consistent if we can go with LocalDateTime in all cases whenever possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r494288466
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PathFilterSuite.scala
##########
@@ -0,0 +1,593 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.File
+import java.time.{LocalDateTime, ZoneOffset}
+import java.time.format.DateTimeFormatter
+
+import scala.util.Random
+
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.util.{stringToFile, DateTimeUtils}
+import org.apache.spark.sql.test.SharedSparkSession
+
+class PathFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ def createSingleFile(dir: File): File = {
+ val file = new File(dir, "temp" + Random.nextInt(1000) + ".csv")
+ stringToFile(file, "text")
+ }
+
+ def setFileTime(time: LocalDateTime, file: File): Boolean = {
+ val sameTime = time.toEpochSecond(ZoneOffset.UTC)
+ file.setLastModified(sameTime * 1000)
+ }
+
+ def setPlusFileTime(time: LocalDateTime, file: File, interval: Long): Boolean = {
+ val sameTime = time.plusDays(interval).toEpochSecond(ZoneOffset.UTC)
+ file.setLastModified(sameTime * 1000)
+ }
+
+ def setMinusFileTime(time: LocalDateTime, file: File, interval: Long): Boolean = {
+ val sameTime = time.minusDays(interval).toEpochSecond(ZoneOffset.UTC)
+ file.setLastModified(sameTime * 1000)
+ }
+
+ def formatTime(time: LocalDateTime): String = {
+ time.format(DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss"))
+ }
+
+ test(
+ "SPARK-31962: when modifiedBefore specified" +
+ " and sharing same timestamp with file last modified time.") {
+ withTempDir { dir =>
+ val file = createSingleFile(dir)
+ val time = LocalDateTime.now(ZoneOffset.UTC)
+ setFileTime(time, file)
+ val formattedTime = formatTime(time)
+
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedBefore", formattedTime)
+ .option("timeZone", "UTC")
+ .format("csv")
+ .load(dir.getCanonicalPath)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test(
+ "SPARK-31962: when modifiedAfter specified" +
+ " and sharing same timestamp with file last modified time.") {
+ withTempDir { dir =>
+ val file = createSingleFile(dir)
+ val time = LocalDateTime.now()
+ setFileTime(time, file)
+ val formattedTime = formatTime(time)
+
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", formattedTime)
+ .option("timeZone", "UTC")
+ .format("csv")
+ .load(dir.getCanonicalPath)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test(
+ "SPARK-31962: when modifiedBefore and modifiedAfter option" +
+ " share same timestamp with file last modified time.") {
+ withTempDir { dir =>
+ val file = createSingleFile(dir)
+ val time = LocalDateTime.now()
+ setFileTime(time, file)
+ val formattedTime = formatTime(time)
+
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", formattedTime)
+ .option("modifiedBefore", formattedTime)
+ .format("csv")
+ .load(dir.getCanonicalPath)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test(
+ "SPARK-31962: when modifiedBefore and modifiedAfter option" +
+ " share same timestamp with earlier file last modified time.") {
+ withTempDir { dir =>
+ val file = createSingleFile(dir)
+ val time = LocalDateTime.now()
+ setMinusFileTime(time, file, 3)
+
+ val formattedTime = formatTime(time)
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", formattedTime)
+ .option("modifiedBefore", formattedTime)
+ .format("csv")
+ .load(dir.getCanonicalPath)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test(
+ "SPARK-31962: when modifiedBefore and modifiedAfter option" +
+ " share same timestamp with later file last modified time.") {
+ withTempDir { dir =>
+ createSingleFile(dir)
+ val time = LocalDateTime.now()
+ val formattedTime = formatTime(time)
+ val msg = intercept[AnalysisException] {
+ spark.read
+ .option("modifiedAfter", formattedTime)
+ .option("modifiedBefore", formattedTime)
+ .format("csv")
+ .load(dir.getCanonicalPath)
+ }.getMessage
+ assert(msg.contains("Unable to infer schema for CSV"))
+ }
+ }
+
+ test("SPARK-31962: when modifiedAfter specified with a past date") {
+ withTempDir { dir =>
+ val file = createSingleFile(dir)
+ file.setLastModified(DateTimeUtils.currentTimestamp())
Review comment:
Shall we do the similar with below test instead of using specific date/format?
Moreover, it would be consistent if we can go with LocalDateTime in all cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672095088
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-644667095
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r440893695
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
##########
@@ -356,26 +380,35 @@ object InMemoryFileIndex extends Logging {
bulkListLeafFiles(
dirs.map(_.getPath),
hadoopConf,
- filter,
+ filters,
session,
- areRootPaths = false
+ areRootPaths = false,
+ parameters = parameters
).flatMap(_._2)
case _ =>
dirs.flatMap { dir =>
listLeafFiles(
dir.getPath,
hadoopConf,
- filter,
+ filters,
sessionOpt,
ignoreMissingFiles = ignoreMissingFiles,
ignoreLocality = ignoreLocality,
- isRootPath = false)
+ isRootPath = false,
+ parameters = parameters)
}
}
+ val fileFilters = (filters ++ FileModifiedDateOption.accept(
+ parameters,
+ sessionOpt.get,
+ hadoopConf))
+ .filter(filter => filter !=null)
+
val allFiles = topLevelFiles ++ nestedFiles
- if (filter != null) allFiles.filter(f => filter.accept(f.getPath)) else allFiles
Review comment:
Once we added the ability to use classes extending from PathFilter, I realized we would only allow one filter to be applied at anytime. Given the number of consumers calling InMemoryFileIndex methods, this seemed unideal, so I extended this to use a parallel array supporting multiple filters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663850911
**[Test build #126546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126546/testReport)** for PR 28841 at commit [`e62568a`](https://github.com/apache/spark/commit/e62568ab07ffcf0dfa760bb09e44cee27e2bddd8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673868046
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673864599
**[Test build #127431 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127431/testReport)** for PR 28841 at commit [`4329c8a`](https://github.com/apache/spark/commit/4329c8abeb64702c5b92880e11b76511087da841).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650098733
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-721756827
**[Test build #130605 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130605/testReport)** for PR 28841 at commit [`f88ef62`](https://github.com/apache/spark/commit/f88ef62ebcf534779e80d590c5f418ccd2818728).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-666732439
> There are quite a bit of commits here. Would it be preferred if I re-opened this PR for a cleaner merge to master?
Let's hear the voice on reopening the PR, as there're already huge number of review comments, and once you reopen the PR these comments are lost. We can do that at any time we feel it's needed, even just before merging, so personally it won't matter much.
> Once this is committed, I'd be curious to follow up on SPARK-32155. I noticed your repo, https://github.com/HeartSaVioR/structured_streaming_experiments, for stress-testing streaming from a file data source. Particularly your findings between streaming from a file data source to delta lake vs Iceberg.
I've raised related discussion thread, but haven't got enough inputs so far. Please add your voice on discussion if you are interested - IMHO latestFirst and start by timestamp are conflicting and can't go together.
https://lists.apache.org/thread.html/r08e3a8d7df74354b38d19ffdebe1afe7fa73c2f611f0a812a867dffb%40%3Cdev.spark.apache.org%3E
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672559298
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-720961219
> Sigh. Now it has conflicts so I can't go with giving +1 and merging.
>
> @cchighman Could you please fix the conflicts? It would be totally OK if you'd like to let me take this over - just wanted to give a full credit of yours and accelerate the process of merging. If you can deal with fixing conflicts and hopefully also address my minor comments I'll give +1 if tests pass.
Ill take care of it now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663921444
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663848363
**[Test build #126544 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126544/testReport)** for PR 28841 at commit [`853c432`](https://github.com/apache/spark/commit/853c4324958e0b756d51b6d9b5c94e17fb6ebb7a).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-652850721
@cchighman Thanks for reading through the huge wall of text!
I agree the option can be provided to batch query only, and consider how to apply the option to structured streaming later (as we don't have solid idea yet). Just to reiterate, I guess we may want to still discuss only lower bound vs lower & upper bound, even in batch case.
I also agree the option can be simply applied to the structured streaming (only for lower bound) on top of current options. That would play as a "filter". As I mentioned in #28422 I already proposed the similar thing, though the purpose was for applying "retention" hence dynamically changing instead of be static.
That said, the problem is, this approach doesn't help to cut down file stream source metadata log, which is another known major issue in file stream source. File stream source remembers every file you processed and never drops anything. My viewpoint is focused on how we can minimize the entries to remember across long query run. We have maxFileAge option but due to some issue (combination with latestFirst) it doesn't help minimizing the entries. It wouldn't be good if we introduce another similar option but leave the major issue behind.
That's why I cannot agree simply to apply the option to SS. It deserves another level of consideration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-650519318
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663848366
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-654007411
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-671989831
**[Test build #127331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127331/testReport)** for PR 28841 at commit [`042c36e`](https://github.com/apache/spark/commit/042c36ed531b0d6e76295b5e9b17889f036a160f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672093114
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672269109
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127341/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663954466
**[Test build #126566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126566/testReport)** for PR 28841 at commit [`f849059`](https://github.com/apache/spark/commit/f849059b626153803b4e44c4d93a6110116672e5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r463961032
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
##########
@@ -561,6 +564,367 @@ class FileBasedDataSourceSuite extends QueryTest
}
}
+ test("SPARK-31962 - when modifiedAfter specified " +
+ "with a past date") {
Review comment:
you don't need the line break.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672661201
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-663929055
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126560/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
cchighman commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-667501680
@HeartSaVioR @gengliangwang @cloud-fan
Gentle ping to see if everything looks good. I feel like the final design after incorporating feedback worked out really well here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-693216812
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-646396968
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-672738044
**[Test build #127375 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127375/testReport)** for PR 28841 at commit [`0e069a7`](https://github.com/apache/spark/commit/0e069a76145db28307de1a32f3ff52691222eb31).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-722330338
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35258/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673862283
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-673191395
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127391/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-652850721
@cchighman Thanks for reading through the huge wall of text!
I agree the option can be provided to batch query only, and consider how to apply the option to structured streaming later (as we don't have solid idea yet). Just to reiterate, I guess we may want to still discuss only lower bound vs lower & upper bound, even in batch case.
I also agree the option can be simply applied to the structured streaming (only for lower bound) on top of current options. That would play as a "filter". As I mentioned in #28422 I already proposed the similar thing, though the purpose was for applying "retention" hence dynamically changing instead of be static.
That said, the problem is, this approach doesn't help to cut down file stream source metadata log, which is another known major issue in file stream source. File stream source remembers every file you processed and never drops anything. My viewpoint is focused on how we can minimize the entries to remember across long query run. We have maxFileAge option but due to some issue it doesn't help minimizing the entries. It wouldn't be good if we introduce another similar option but leave the major issue behind.
That's why I cannot agree simply to apply the option to SS. It deserves another level of consideration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28841:
URL: https://github.com/apache/spark/pull/28841#issuecomment-680018752
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org