[GitHub] [gobblin] umustafi commented on a diff in pull request #3656: [GOBBLIN-1797] Skip scheduling flows far into future

["umustafi (via GitHub)" <gi...@apache.org>]

umustafi commented on code in PR #3656:
URL: https://github.com/apache/gobblin/pull/3656#discussion_r1130141949


##########
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java:
##########
@@ -90,7 +90,9 @@ public class ConfigurationKeys {
   public static final String JOB_RETRIGGERING_ENABLED = "job.retriggering.enabled";
   public static final String DEFAULT_JOB_RETRIGGERING_ENABLED = "true";
   public static final String LOAD_SPEC_BATCH_SIZE = "load.spec.batch.size";
-  public static final int DEFAULT_LOAD_SPEC_BATCH_SIZE = 100;
+  public static final int DEFAULT_LOAD_SPEC_BATCH_SIZE = 500;
+  public static final String SKIP_SCHEDULING_FLOWS_AFTER_NUM_DAYS = "skip.scheduling.flows.after.num.days";
+  public static final int DEFAULT_NUM_DAYS_TO_SKIP_AFTER = 100;

Review Comment:
   In reality there's almost no case people will schedule in advance of more than a month out, maybe some monthly flow or weekly but more likely it's daily. Majority of these flows should actually be either years out (2050) or well within this time frame. What do you think about that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org