[GitHub] [flink] lvhuyen commented on a change in pull request #6613: [FLINK-9940] [API/DataStream][File source] out-of-order files were missed in continuous monitoring

[GitBox <gi...@apache.org>]

lvhuyen commented on a change in pull request #6613:
URL: https://github.com/apache/flink/pull/6613#discussion_r484131667



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/ContinuousFileMonitoringFunction.java
##########
@@ -93,20 +102,45 @@
 	/** Which new data to process (see {@link FileProcessingMode}. */
 	private final FileProcessingMode watchType;
 
-	/** The maximum file modification time seen so far. */
+	/** The offset interval back from the latest file modification timestamp to scan for our-of-order files.
+	 *  Valid value for this is from 0 to Long.MAX_VALUE.
+	 *
+	 *  <p><b>NOTE: </b>: Files with (modTime > Long.MIN_VALUE + readConsistencyOffset) will NOT be read.
+	 */
+	private final long readConsistencyOffset;
+
+	/** The current modification time watermark. */
 	private volatile long globalModificationTime = Long.MIN_VALUE;
 
+	/** The maximum file modification time seen so far. */
+	private volatile long maxProcessedTime = Long.MIN_VALUE;

Review comment:
       There has been a recent change which added the initialization for globalModificationTime in the constructor, thus I will be making the change for that variable as well.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org