[GitHub] [spark] linehrr opened a new pull request #24548: [DOCS] Added missing config `maxFileAge` in file streaming source in SS

[GitBox <gi...@apache.org>]

linehrr opened a new pull request #24548: [DOCS] Added missing config `maxFileAge` in file streaming source in SS
URL: https://github.com/apache/spark/pull/24548
 
 
   ## What changes were proposed in this pull request?
   
   added the missing config for structured streaming when using file source.
   
   from the code we have 
   ```
     /**
      * Maximum age of a file that can be found in this directory, before it is ignored. For the
      * first batch all files will be considered valid. If `latestFirst` is set to `true` and
      * `maxFilesPerTrigger` is set, then this parameter will be ignored, because old files that are
      * valid, and should be processed, may be ignored. Please refer to SPARK-19813 for details.
      *
      * The max age is specified with respect to the timestamp of the latest file, and not the
      * timestamp of the current system. That this means if the last file has timestamp 1000, and the
      * current system time is 2000, and max age is 200, the system will purge files older than
      * 800 (rather than 1800) from the internal state.
      *
      * Default to a week.
      */
     val maxFileAgeMs: Long =
       Utils.timeStringAsMs(parameters.getOrElse("maxFileAge", "7d"))
   ```
   which is not documented. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org