[jira] [Work logged] (GOBBLIN-1797) Skip scheduling flows far into future

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GOBBLIN-1797?focusedWorklogId=849707&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-849707 ]

ASF GitHub Bot logged work on GOBBLIN-1797:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 08/Mar/23 01:30
            Start Date: 08/Mar/23 01:30
    Worklog Time Spent: 10m 
      Work Description: umustafi opened a new pull request, #3656:
URL: https://github.com/apache/gobblin/pull/3656

   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [X] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-1797
   
   ### Description
   - [X] Here are some details about my PR, including screenshots (if applicable):
   The [unschedule feature](https://jarvis.corp.linkedin.com/codesearch/result/?name=FlowConfigResourceLocalHandler.java&path=gobblin-elr%2Fgobblin-restli%2Fgobblin-flow-config-service%2Fgobblin-flow-config-service-server%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fservice&reponame=linkedin%2Fgobblin-elr#62) sets a schedule to run Jan 1st of 2050 so far in advance that it will "never run" but potentially there are over 100k of these flows so we are loading and scheduling many unnecessary flows increasing the startup time. On initialization we add a check that verifies the next run of the flow is within a certain time frame (100 days by default) and loads it into the scheduler if it is within that time frame. We choose that default value under the assumption that we will redeploy GaaS at least every 100 days and then if we approach a far out scheduled flow we will load it into the Scheduler. However, in most cases uses schedule flows for near future or immediately and those will all be scheduled. This PR also renames metrics and adds helpful new ones. 
   
   ### Tests
   - [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   Unit test checks the functionality for calculating the number of days until the next run for a cron expression and evaluates cases of far out scheduled flows, regularly occurring, and blank flows (adhoc ones).
   
   ### Commits
   - [X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   




Issue Time Tracking
-------------------

            Worklog Id:     (was: 849707)
    Remaining Estimate: 0h
            Time Spent: 10m

> Skip scheduling flows far into future
> -------------------------------------
>
>                 Key: GOBBLIN-1797
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1797
>             Project: Apache Gobblin
>          Issue Type: Improvement
>          Components: gobblin-service
>            Reporter: Urmi Mustafi
>            Assignee: Abhishek Tiwari
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> The unschedule feature linked below sets a schedule to run Jan 1st of 2050 so far in advance that it will "never run" [https://jarvis.corp.linkedin.com/codesearch/result/?name=FlowConfigResourceLocalHandler.java&path=gobblin-elr%2Fgobblin-restli%2Fgobblin-flow-config-service%2Fgobblin-flow-config-service-server%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fservice&reponame=linkedin%2Fgobblin-elr#62] but potentially there are over 100k of these flows so we are loading and scheduling many unnecessary flows. On initialization we add a check that verifies the next run of the flow is within a certain time frame (100 days by default) and loads it into the scheduler if it is within that time frame. We choose that default value under the assumption that we will redeploy GaaS at least every 100 days and then if we approach a far out scheduled flow we will load it into the Scheduler. However, in most cases uses schedule flows for near future or immediately and those will all be scheduled. This PR also renames metrics and adds helpful new ones. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)