You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/04/30 12:17:35 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
HeartSaVioR opened a new pull request #28422:
URL: https://github.com/apache/spark/pull/28422
### What changes were proposed in this pull request?
This PR introduces a new option, `inputRetention` to provide a way to specify retention on input files.
`maxAgeMs` plays as `soft` limit (it doesn't apply for some conditions like first batch, as well as it's applied relatively to the modified time of input files). Given it's not consistently applied across the matrix of configurations, Spark cannot purge the entries based on the configuration. (Streaming query can change the configurations and be relaunched.)
`inputRetention` plays as `hard` limit - Spark will not include files older than the retention as input files, as well as tries to exclude file entries older than the retention (it actually happens on compaction, as it's the only phase to remove entries).
`inputRetention` is relative to the system timestamp unlike `maxAgeMs`, which is easier for end users to reason about. This would require end users to correctly set the nodes' timestamp, but in most cases they would do it in other reasons as well. Also, this would filter out old files when the query intends to replay from input files, hence this should be considered as well.
### Why are the changes needed?
This has been a pain to deal with metadata growing in both file stream source and file stream sink. For file stream source, all processed input files are tracked which size is continuously growing, and there's no approach on reducing the size/entries. In compact batch, it reads all previous input files to write new compact file, which brings major latency.
The issue is even reported from user group, refer here: https://lists.apache.org/thread.html/r897771f5526d10d0b13da9177a6b7d2e378888b22823c839cceea457%40%3Cuser.spark.apache.org%3E
### Does this PR introduce _any_ user-facing change?
This doesn't bring any change "by default", as the new configuration is optional. (The default value is set to unrealistic one making it effectively none.)
This adds a new configuration - previous sections described the behavior.
### How was this patch tested?
New UTs verifying two behaviors per test.
1) old files should not be included as input files if input retention is specified
2) when compacting, outdated entries should be filtered out
I've manually tested with above two behaviors as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643728308
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123985/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-641097576
I agree the new addition of the similar option feels tricky.
Maybe you've already indicated there're some cases `maxFileAge` has to be ignored which means Spark is never able to drop entries from metadata (e.g. when `latestFirst` is true and `maxFilesPerTrigger` is set). Given all of these options can be changed for the further runs, I was confused whether it'd be safe to drop entries based on the current set of options and status of entries. There looked to be an edge-case input files can be processed more than once.
Also I felt it's less intuitive to reason about the way how the max age is specified - it is with respect to the timestamp of the latest file being figured out from Spark, not the timestamp of the current system. (But well... That might be only me.)
The new option ensures that the behavior is consistent regardless of these options. It just plays as "hard" limit and in any case Spark won't handle the files which are older than the threshold. (Suppose these files are simply deleted due to the retention policy - not physically though) It applies on both forward read and backward read, doesn't matter how many files Spark will read in a batch.
(Personally, I think `maxFileAge` itself should work like the way, and then we wouldn't have such confusion.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675301889
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675297171
**[Test build #127531 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127531/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-659390230
**[Test build #125975 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125975/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643706117
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692515053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128682/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643706501
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28422:
URL: https://github.com/apache/spark/pull/28422#discussion_r418381154
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -542,6 +542,12 @@ Here are the details of all the sources in Spark.
<br/>
<code>maxFileAge</code>: Maximum age of a file that can be found in this directory, before it is ignored. For the first batch all files will be considered valid. If <code>latestFirst</code> is set to `true` and <code>maxFilesPerTrigger</code> is set, then this parameter will be ignored, because old files that are valid, and should be processed, may be ignored. The max age is specified with respect to the timestamp of the latest file, and not the timestamp of the current system.(default: 1 week)
<br/>
+ <code>inputRetention</code>: Maximum age of a file that can be found in this directory, before it is ignored.<br/>
+ This is the "hard" limit of input data retention - input files older than the max age will be ignored regardless of source options (while `maxFileAgeMs` depends on the condition), as well as entries in checkpoint metadata will be purged based on this.<br/>
+ Unlike `maxFileAgeMs`, the max age is specified with respect to the timestamp of the current system, to provide consistent behavior regardless of metadata entries.<br/>
+ NOTE 1: Please be careful to set the value if the query replays from the old input files.<br/>
+ NOTE 2: Please make sure the timestamp is in sync between nodes which run the query.<br/>
+ <br/>
Review comment:
Out of curiosity, did we document what kind of value is expected for these options, e.g., `7d`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-659590381
**[Test build #125975 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125975/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-657318840
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125724/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751138195
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675301381
**[Test build #127547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127547/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-640538971
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643757806
**[Test build #123998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123998/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-621811759
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-640424887
> Maybe the user group reference can be added as a comment and not as commit msg.
I'll move it to the JIRA description - personally I'd be OK to leave the rationalization in commit message, but I agree it's redundant and make commit message be verbose.
(EDIT: I already did that, skipping)
>> I've manually tested with above two behaviors as well.
> How do you mean that exactly?
I meant both two behaviors are manually tested
1. old files should not be included as input files if input retention is specified
2. when compacting, outdated entries should be filtered out
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-775198833
**[Test build #135034 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135034/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-659390949
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-659388927
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-657271684
**[Test build #125724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125724/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643706409
**[Test build #123985 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123985/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-644757113
I agree, confusion comes from `latestFirst` basically.
> But then should we really open the possibility to trace back older files?
I see a use-case where it's useful. The query is has fallen behind and files have piled up. The query must keep-up with the incoming data but also must process older files as a side job.
> Would we just simply do the thing we do with Kafka's "latest" option, which only affects the first batch and no-op in further batches?
Not sure how exactly `latestFirst` should behave then?! Create a single gigantic micro-batch which processes all the data and then switch back to normal mode?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-640431340
**[Test build #123621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123621/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28422:
URL: https://github.com/apache/spark/pull/28422#discussion_r418379475
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
##########
@@ -71,8 +71,13 @@ class FileStreamSource(
Map()
}}
+ /** exposed for testing */
+ private[sql] var clockForRetention: Clock = new SystemClock
Review comment:
Let's remove `private[sql]` because `execution` is already a private package (SPARK-16964)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-622764099
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634495624
**[Test build #123172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123172/testReport)** for PR 28422 at commit [`2af1df1`](https://github.com/apache/spark/commit/2af1df14a9efc58162402d6e76718fc226a7970d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-661645101
FYI, I've initiated the discussion around this in dev@ mailing list, how to deal with "latestFirst" option and metadata growing.
https://lists.apache.org/thread.html/r08e3a8d7df74354b38d19ffdebe1afe7fa73c2f611f0a812a867dffb%40%3Cdev.spark.apache.org%3E
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692524467
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751622016
**[Test build #133435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133435/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675297527
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634632846
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634307438
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634471108
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634305928
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692515044
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634471108
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634421967
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-640431800
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634421967
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751308881
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133389/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-640541944
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675297540
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127531/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-622942583
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-657271786
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751620885
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-622762782
**[Test build #122203 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122203/testReport)** for PR 28422 at commit [`2af1df1`](https://github.com/apache/spark/commit/2af1df14a9efc58162402d6e76718fc226a7970d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #28422:
URL: https://github.com/apache/spark/pull/28422#discussion_r436026041
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -542,6 +542,12 @@ Here are the details of all the sources in Spark.
<br/>
<code>maxFileAge</code>: Maximum age of a file that can be found in this directory, before it is ignored. For the first batch all files will be considered valid. If <code>latestFirst</code> is set to `true` and <code>maxFilesPerTrigger</code> is set, then this parameter will be ignored, because old files that are valid, and should be processed, may be ignored. The max age is specified with respect to the timestamp of the latest file, and not the timestamp of the current system.(default: 1 week)
<br/>
+ <code>inputRetention</code>: Maximum age of a file that can be found in this directory, before it is ignored. (e.g. 14d, default: None)<br/>
+ This is the "hard" limit of input data retention - input files older than the max age will be ignored regardless of source options (while `maxFileAgeMs` depends on the condition), as well as entries in checkpoint metadata will be purged based on this.<br/>
+ Unlike `maxFileAgeMs`, the max age is specified with respect to the timestamp of the current system, to provide consistent behavior regardless of metadata entries.<br/>
+ NOTE 1: Please be careful to set the value if the query replays from the old input files.<br/>
+ NOTE 2: Please make sure the timestamp is in sync between nodes which run the query.<br/>
+ <br/>
"file:///dataset.txt"<br/>
Review comment:
While I was reviewing this I've pinpointed a parameter description split and opened https://github.com/apache/spark/pull/28739
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692523805
**[Test build #128702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128702/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675250864
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-622001606
**[Test build #122127 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122127/testReport)** for PR 28422 at commit [`738caa1`](https://github.com/apache/spark/commit/738caa19d7be6bcc6612c1e409a5c548b0b987c7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643729486
**[Test build #123998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123998/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643729486
**[Test build #123998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123998/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643729336
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634492294
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-621807517
**[Test build #122127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122127/testReport)** for PR 28422 at commit [`738caa1`](https://github.com/apache/spark/commit/738caa19d7be6bcc6612c1e409a5c548b0b987c7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-657271786
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643706409
**[Test build #123985 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123985/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675297527
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692515044
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751308450
**[Test build #133389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133389/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-640539460
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-679867829
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-657271684
**[Test build #125724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125724/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675250414
**[Test build #127531 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127531/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692387211
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634307438
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-659592354
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-775221213
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135034/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643271165
I've analyzed this further. I have the same opinion about `maxFileAge`, namely it's unintuitive how it's programmed. I think it should be like:
* `maxFileAge` should behave like `inputRetention`. Retention is based on current timestamp normally. We shouldn't go far, Kafka and similar components does that.
* The current feature should depend on `maxFileAge`
If the user wants to operate a query with `latestFirst` in long term then I see these options:
* User sets `maxFileAge` properly => no file loss just some fluctuation in the number of not processed files
* User doesn't set `maxFileAge` properly but cluster sized properly => configuration issue, because with proper value all the files must be processed within `maxFileAge`.
* User doesn't set `maxFileAge` properly and cluster sized badly => sizing and configuration issue. Cluster computation power must be increased to have room for the old not yet processed files. As in the previous case choosing appropriate `maxFileAge` is important.
The last point can be problematic and can end-up in data loss but this is exactly the same when processing data from Kafka. If retention fires then the data just disappear w/o any notification. This situation is better though because if the query is not able to catch-up then it can be restarted with bigger `maxFileAge` and cluster, allowing the query to catch up properly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675250207
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-621811759
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-679993860
**[Test build #127882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127882/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751308881
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133389/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751623594
**[Test build #133435 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133435/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634470901
**[Test build #123161 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123161/testReport)** for PR 28422 at commit [`2af1df1`](https://github.com/apache/spark/commit/2af1df14a9efc58162402d6e76718fc226a7970d).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-679994900
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692389547
**[Test build #128682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128682/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-640431340
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-657318834
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-640431800
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751623610
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-622002774
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-622002774
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643728212
**[Test build #123985 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123985/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675442946
**[Test build #127547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127547/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751625093
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38026/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634421190
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-622942583
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-621796985
cc. @tdas @zsxwing @jose-torres @gaborgsomogyi
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643705743
I can even tolerate the fact maxFileAge is originated from path's latest timestamp. If we don't believe the node's wall time (I suspect other logic works well in such case though) then yes it might be the source of the truth across nodes.
I feel all the confusions come from the behavior of `latestFirst`. Yes we would like to read from latest in some case if we're only interested with latest files. But then should we really open the possibility to trace back older files? Would we just simply do the thing we do with Kafka's "latest" option, which only affects the first batch and no-op in further batches?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692523314
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692695897
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-775195497
**[Test build #135034 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135034/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634471117
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123161/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692387695
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28422:
URL: https://github.com/apache/spark/pull/28422#discussion_r418571213
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -542,6 +542,12 @@ Here are the details of all the sources in Spark.
<br/>
<code>maxFileAge</code>: Maximum age of a file that can be found in this directory, before it is ignored. For the first batch all files will be considered valid. If <code>latestFirst</code> is set to `true` and <code>maxFilesPerTrigger</code> is set, then this parameter will be ignored, because old files that are valid, and should be processed, may be ignored. The max age is specified with respect to the timestamp of the latest file, and not the timestamp of the current system.(default: 1 week)
<br/>
+ <code>inputRetention</code>: Maximum age of a file that can be found in this directory, before it is ignored.<br/>
+ This is the "hard" limit of input data retention - input files older than the max age will be ignored regardless of source options (while `maxFileAgeMs` depends on the condition), as well as entries in checkpoint metadata will be purged based on this.<br/>
+ Unlike `maxFileAgeMs`, the max age is specified with respect to the timestamp of the current system, to provide consistent behavior regardless of metadata entries.<br/>
+ NOTE 1: Please be careful to set the value if the query replays from the old input files.<br/>
+ NOTE 2: Please make sure the timestamp is in sync between nodes which run the query.<br/>
+ <br/>
Review comment:
Looks like the kinds of values weren't specified in many options, but implied by default values. This option doesn't have default value - maybe better to explicitly specify kind of value. Good point!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-639540113
Just picked this up. Maybe the user group reference can be added as a comment and not as commit msg.
> I've manually tested with above two behaviors as well.
How do you mean that exactly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-844581528
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #28422:
URL: https://github.com/apache/spark/pull/28422
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-622762782
**[Test build #122203 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122203/testReport)** for PR 28422 at commit [`2af1df1`](https://github.com/apache/spark/commit/2af1df14a9efc58162402d6e76718fc226a7970d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-679994900
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-679867421
**[Test build #127882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127882/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751308973
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37980/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692389547
**[Test build #128682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128682/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751308973
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37980/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi edited a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi edited a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-644757113
I agree, confusion comes from `latestFirst` basically.
> But then should we really open the possibility to trace back older files?
I see a use-case where it's useful. The query has has fallen behind and files have piled up. The query must keep-up with the incoming data but also must process older files as a side job.
> Would we just simply do the thing we do with Kafka's "latest" option, which only affects the first batch and no-op in further batches?
Not sure how exactly `latestFirst` should behave then?! Create a single gigantic micro-batch which processes all the data and then switch back to normal mode?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-679866088
Just rebased. I haven't reached consensus around the discussion in dev mailing list though. I'll bump it again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634306965
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-657271083
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692523805
**[Test build #128702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128702/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675298909
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-657318411
**[Test build #125724 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125724/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634421709
**[Test build #123161 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123161/testReport)** for PR 28422 at commit [`2af1df1`](https://github.com/apache/spark/commit/2af1df14a9efc58162402d6e76718fc226a7970d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #28422:
URL: https://github.com/apache/spark/pull/28422
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751623610
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133435/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675444581
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692512307
**[Test build #128682 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128682/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-640424887
> Maybe the user group reference can be added as a comment and not as commit msg.
I'll move it to the JIRA description - personally I'd be OK to leave the rationalization in commit message, but I agree it's redundant and make commit message be verbose.
>> I've manually tested with above two behaviors as well.
> How do you mean that exactly?
I meant both two behaviors are manually tested
1. old files should not be included as input files if input retention is specified
2. when compacting, outdated entries should be filtered out
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-621807517
**[Test build #122127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122127/testReport)** for PR 28422 at commit [`738caa1`](https://github.com/apache/spark/commit/738caa19d7be6bcc6612c1e409a5c548b0b987c7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643758026
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-640539460
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751308450
**[Test build #133389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133389/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634632846
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643706501
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643758026
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-657318834
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi edited a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi edited a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-644757113
I agree, confusion comes from `latestFirst` basically.
> But then should we really open the possibility to trace back older files?
I see a use-case where it's useful. The query has has fallen behind and files have piled up. The query must keep-up with the incoming data but also must process older files as a side job. Catching up a situation where all the old files are processed can be a super slow thing. That's the reason why I see a value to split it up with `maxFilesPerTrigger`.
> Would we just simply do the thing we do with Kafka's "latest" option, which only affects the first batch and no-op in further batches?
Not sure how exactly `latestFirst` should behave then?! Create a single gigantic micro-batch which processes all the data and then switch back to normal mode?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675250864
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675301889
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634306965
**[Test build #123141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123141/testReport)** for PR 28422 at commit [`2af1df1`](https://github.com/apache/spark/commit/2af1df14a9efc58162402d6e76718fc226a7970d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-659390230
**[Test build #125975 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125975/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634421709
**[Test build #123161 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123161/testReport)** for PR 28422 at commit [`2af1df1`](https://github.com/apache/spark/commit/2af1df14a9efc58162402d6e76718fc226a7970d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675444581
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-659592354
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643728307
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-622764099
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643728307
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692387703
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692524467
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692694872
**[Test build #128702 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128702/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634631767
**[Test build #123172 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123172/testReport)** for PR 28422 at commit [`2af1df1`](https://github.com/apache/spark/commit/2af1df14a9efc58162402d6e76718fc226a7970d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-622942299
**[Test build #122203 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122203/testReport)** for PR 28422 at commit [`2af1df1`](https://github.com/apache/spark/commit/2af1df14a9efc58162402d6e76718fc226a7970d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643729592
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675301381
**[Test build #127547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127547/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-679867829
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634492294
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-692695897
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-775221213
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135034/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-679867421
**[Test build #127882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127882/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751308879
**[Test build #133389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133389/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-751622016
**[Test build #133435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133435/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-643729592
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634495624
**[Test build #123172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123172/testReport)** for PR 28422 at commit [`2af1df1`](https://github.com/apache/spark/commit/2af1df14a9efc58162402d6e76718fc226a7970d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-634491641
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-659390949
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-675250414
**[Test build #127531 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127531/testReport)** for PR 28422 at commit [`06ee53d`](https://github.com/apache/spark/commit/06ee53d9dee60756be8563d584d589e198d670f1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28422:
URL: https://github.com/apache/spark/pull/28422#issuecomment-775195497
**[Test build #135034 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135034/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org