You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "fengjian1129 (via GitHub)" <gi...@apache.org> on 2024/10/18 04:02:31 UTC

[I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

fengjian1129 opened a new issue, #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706

   ### Search before asking
   
   - [X] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.
   
   
   ### Motivation
   
   DolphinScheduler currently lacks dependency types scheduled at the same cycle level, such as the current `thisMonth`  type, which mainly detects instances that have been successfully executed every day of the current month before they can continue to execute. In most data generation, two tasks scheduled at the same month level are also required, and only instances that have been successfully executed once in the current month need to be detected. However, other scheduling dependency types also have similar issues, such as `lastMonth`,`lastWeek`,`last7Days`, `last3Days`, etc. They are all task dependency types that detect successful instances within a certain time range. We need a task dependency type that only detects one successful instance within the execution cycle.
   
   ### Design Detail
   
   Add cycle dependency type , such as `last1Date` `last2Date` ... `lastNdate` `lastNmonthDate` `lastNWeek`,etc.
   only detects one successful instance within the execution cycle.
   
   ### Compatibility, Deprecation, and Migration Plan
   
   _No response_
   
   ### Test Plan
   
   _No response_
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "fengjian1129 (via GitHub)" <gi...@apache.org>.
fengjian1129 commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2421871618

   > > May I ask if there is any documentation for compiling backend code and frontend code? I want to try debugging, I only submitted backend code before
   > 
   > You can take a look at https://dolphinscheduler.apache.org/zh-cn/docs/dev/contribute/development-environment-setup
   
   thx , bro 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2421448942

   What does the UI look like when lastNdate is selected? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2421674025

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2421861527

   > May I ask if there is any documentation for compiling backend code and frontend code? I want to try debugging, I only submitted backend code before
   
   You can take a look at https://dolphinscheduler.apache.org/zh-cn/docs/dev/contribute/development-environment-setup


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "fengjian1129 (via GitHub)" <gi...@apache.org>.
fengjian1129 commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2421870098

   > https://dolphinscheduler.apache.org/zh-cn/docs/dev/contribute/development-environment-setup
   
   thx, bro


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "dill21yu (via GitHub)" <gi...@apache.org>.
dill21yu commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2909207916

       Add a new time-period-based result calculation method. Returns success if any workflow instance succeeds within the period (similar to calculateResultForTasks).
   private DependResult calculateResultForTasks(DependentItem dependentItem,
                                                    List<DateInterval> dateIntervals,
                                                    int testFlag) {
   
           DependResult result = DependResult.FAILED;
           for (DateInterval dateInterval : dateIntervals) {
               ProcessInstance processInstance = findLastProcessInterval(dependentItem.getDefinitionCode(),
                       dateInterval, testFlag);
               if (processInstance == null) {
                   return DependResult.WAITING;
               }
               // need to check workflow for updates, so get all task and check the task state
               if (dependentItem.getDepTaskCode() == Constants.DEPENDENT_WORKFLOW_CODE) {
                   result = dependResultByProcessInstance(processInstance);
               } else if (dependentItem.getDepTaskCode() == Constants.DEPENDENT_ALL_TASK_CODE) {
                   result = dependResultByAllTaskOfProcessInstance(processInstance, testFlag);
               } else {
                   result = dependResultBySingleTaskInstance(processInstance, dependentItem.getDepTaskCode(), testFlag);
               }
               if (result != DependResult.SUCCESS) {
                   break;
               }
           }
           return result;
       }


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2969177088

   > [@SbloodyS](https://github.com/SbloodyS) [@ruanwenjun](https://github.com/ruanwenjun) Help to see if it is feasible to make changes to the dependency cycle type?
   
   Basically LGTM, we may need to use a dynamically expression e.g. `(T-1,h)`, `(T-1, d)` to expression this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-3003564674

   > The new 'thisMonthPeriod' means that the dependency condition only needs to satisfy any successful instance within this month.
   
   It's not. `thisMonthPeriod` to represent a continuous month. It means that every day of this month is consistent with the current dependence.
   
   > Besides 'thisMonthPeriod' and 'last1MonthPeriod', do we also need to add types like 'last1WeekPeriod' or others? 
   
   Yes. We also need `lastNMonthPeriod`.
   
   > Can this issue be assigned to me to work on
   
   Sure. We need to discuss it clearly first before PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "dill21yu (via GitHub)" <gi...@apache.org>.
dill21yu commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2961390824

   @SbloodyS @ruanwenjun Help to see if it is feasible to make changes to the dependency cycle type?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "dill21yu (via GitHub)" <gi...@apache.org>.
dill21yu commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2909174198

   We also encountered this issue in practical use. It can be resolved by introducing a time period type, such as anyDayLastMonth, ensuring that downstream dependent tasks can function properly as long as the workflow instance was successfully executed at least once during that period. I am willing to submit a PR ![Image](https://github.com/user-attachments/assets/4c55f94b-f7ed-4c6d-8291-20a40f564a95)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2972551009

   > Add several types:"anyTimeToday,anyTimeLast1Days,anyTimeLast2Days,anyTimeLast3Days,anyTimeLast7Days,anyTimeThisWeek,anyTimeLastWeek,anyTimeThisMonth,anyTimeLastMonth"
   Add a parameter in calculateResultForTasks to determine if the task execution result is "any time in a period of time". If any workflow instance is successful during the time period, it will return a success status
   
   At present, we have two different types of dependency cycles.
   1. Individual cycle dependence, such as one hour, last day, etc...
   2. Continuous time dependence, for example, last month means a month in a row.
   
   Adding anytime types directly to the status quo will lead to confusion among users. So I suggest that the naming methods of these types be combed again.
   1. Individual cycle dependence 
   1.1 `currentHour` stay as it is.
   1.2 Using `lastNHours` to represent any hours and use expression to parse it.
   1.3 `lastWeek` stay as it is.
   1.4 `lastMonday` stay as it is.
   1.5 `lastTuesday` stay as it is.
   1.6 `lastWednesday` stay as it is.
   1.7 `lastThursday` stay as it is.
   1.8 `lastFriday` stay as it is.
   1.9 `lastSaturday` stay as it is.
   1.10 `lastSunday` stay as it is.
   1.11 `thisMonth` newly added.
   1.12 Using `lastNMonth` to represent any month and use expression to parse it.
   2. Continuous time dependence
   2.1 `thisMonthPeriod` to represent a continuous month.
   2.2 `last1MonthPeriod` to represent a specified continuous month and use expression to parse it.
   
   This is just my initial idea. If you have a better idea, please feel free to point it out. @dill21yu 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "dill21yu (via GitHub)" <gi...@apache.org>.
dill21yu commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-3279782690

   Could you let me know if this approach looks good? @SbloodyS @ruanwenjun 
   add a checkMode field to differentiate between “ANY_SUCCESS” and “ALL_SUCCESS” requirements while maintaining backward compatibility.
   <html>
   <body>
   <!--StartFragment--><p style="line-height: 1.5; margin-top: 0px; color: rgb(55, 65, 81); font-family: &quot;Microsoft YaHei&quot;, 微软雅黑, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>Behavior Examples</strong>:</p>
   dateValue | checkMode | Meaning | Legacy Equivalent
   -- | -- | -- | --
   last7Days | ANY_SUCCESS | Any single success in 7 days (default) | Original anyTimeLast7Days
   last7Days | ALL_SUCCESS | Requires success every day in 7d | N/A (new)
   last1Month | ANY_SUCCESS | Any success in the last month | Original lastMonth
   
   <!--EndFragment-->
   </body>
   </html>
   
   Advantages:
   Full backward compatibility: Old configs default to ANY_SUCCESS.
   Clear separation: type = time window, checkMode = validation logic.
   Scalable: Future time units (e.g., lastNHours) need no duplicate types.
   
   Why no _Period suffix types? Avoids redundancy (e.g., last7Days + last7DaysPeriod).
   Default ANY_SUCCESS preserves legacy behavior.
   Thanks for the discussion!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "fengjian1129 (via GitHub)" <gi...@apache.org>.
fengjian1129 commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2421499375

   > What does the UI look like when lastNdate is selected?
   
   Now I want to implement some fixed types first because I can't write the front-end interface. I can organize the documents for the newly added fixed scheduling types and submit them to the community


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "dill21yu (via GitHub)" <gi...@apache.org>.
dill21yu commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2995190427

   > > Add several types:"anyTimeToday,anyTimeLast1Days,anyTimeLast2Days,anyTimeLast3Days,anyTimeLast7Days,anyTimeThisWeek,anyTimeLastWeek,anyTimeThisMonth,anyTimeLastMonth"
   > > Add a parameter in calculateResultForTasks to determine if the task execution result is "any time in a period of time". If any workflow instance is successful during the time period, it will return a success status
   > 
   > At present, we have two different types of dependency cycles.
   > 
   > 1. Individual cycle dependence, such as one hour, last day, etc...
   > 2. Continuous time dependence, for example, last month means a month in a row.
   > 
   > Adding anytime types directly to the status quo will lead to confusion among users. So I suggest that the naming methods of these types be combed again.
   > 
   > 1. Individual cycle dependence
   >    1.1 `currentHour` stay as it is.
   >    1.2 Using `lastNHours` to represent any hours and use expression to parse it.
   >    1.3 `lastWeek` stay as it is.
   >    1.4 `lastMonday` stay as it is.
   >    1.5 `lastTuesday` stay as it is.
   >    1.6 `lastWednesday` stay as it is.
   >    1.7 `lastThursday` stay as it is.
   >    1.8 `lastFriday` stay as it is.
   >    1.9 `lastSaturday` stay as it is.
   >    1.10 `lastSunday` stay as it is.
   >    1.11 `thisMonth` newly added.
   >    1.12 Using `lastNMonth` to represent any month and use expression to parse it.
   > 2. Continuous time dependence
   >    2.1 `thisMonthPeriod` to represent a continuous month.
   >    2.2 `last1MonthPeriod` to represent a specified continuous month and use expression to parse it.
   > 
   > This is just my initial idea. If you have a better idea, please feel free to point it out. [@dill21yu](https://github.com/dill21yu)
   
   I agree with your idea. The new 'thisMonthPeriod' means that the dependency condition only needs to satisfy any successful instance within this month. Besides 'thisMonthPeriod' and 'last1MonthPeriod', do we also need to add types like 'last1WeekPeriod' or others? Can this issue be assigned to me to work on? @SbloodyS 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2421634944

   I think this issue proposal is very good. I'm +1 on this.
   
   > Now I want to implement some fixed types first because I can't write the front-end interface. I can organize the documents for the newly added fixed scheduling types and submit them to the community
   
   Since this kind of modification is not very huge. Only when the front-end and back-end as well as the documents and tests are complete will the merger be allowed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-77][task-api] Add cycle dependency type [dolphinscheduler]

Posted by "fengjian1129 (via GitHub)" <gi...@apache.org>.
fengjian1129 commented on issue #16706:
URL: https://github.com/apache/dolphinscheduler/issues/16706#issuecomment-2421783927

   > I think this proposal is very good. I'm +1 on this.
   > 
   > > Now I want to implement some fixed types first because I can't write the front-end interface. I can organize the documents for the newly added fixed scheduling types and submit them to the community
   > 
   > Since this kind of modification is not very huge. Only when the front-end and back-end as well as the documents and tests are complete will the merger be allowed.
   
   May I ask if there is any documentation for compiling backend code and frontend code? I want to try debugging, I only submitted backend code before


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org