[GitHub] [beam] AnandInguva commented on a diff in pull request #28206: Feature/add warnings to file watchers

["AnandInguva (via GitHub)" <gi...@apache.org>]

AnandInguva commented on code in PR #28206:
URL: https://github.com/apache/beam/pull/28206#discussion_r1309360401


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java:
##########
@@ -509,12 +509,22 @@ public MatchConfiguration withEmptyMatchTreatment(EmptyMatchTreatment treatment)
      * condition is reached, where the input to the condition is the filepattern.
      *
      * <p>If {@code matchUpdatedFiles} is set, also watches for files with timestamp change, with
-     * the watching frequency given by the {@code interval}. The pipeline will throw a {@code
-     * RuntimeError} if timestamp extraction for the matched file has failed, suggesting the
+     * the watching frequency given by the {@code interval}. The pipeline will throw a
+     * {@code RuntimeError} if timestamp extraction for the matched file has failed, suggesting the
      * timestamp metadata is not available with the IO connector.
+     *
+     * <p>
+     * Matching continuously scales poorly, as it is stateful, and requires storing file ids in
+     * memory. In addition, because it is memory-only, if a pipeline is restarted, already processed
+     * files will be reprocessed. Consider an alternate technique, such as
+     * <a href="https://cloud.google.com/storage/docs/pubsub-notifications">Pub/Sub Notifications</a>
+     * when using GCS if possible.
+     * </p>
      */
     public MatchConfiguration continuously(
         Duration interval, TerminationCondition<String, ?> condition, boolean matchUpdatedFiles) {
+      LOG.warn("Matching Continuously is stateful, and can scale poorly. Consider using Pub/Sub "

Review Comment:
   I think this is true for python sdk as well. If yes, can we add a warning there?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org