You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by GitBox <gi...@apache.org> on 2022/07/15 00:48:24 UTC
[GitHub] [incubator-devlake] asterix314 opened a new issue, #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
asterix314 opened a new issue, #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
The requested feature is an option to skip failed stages and carry on with the following stages in the pipeline, for those who have a long list of repos to run and want to make as much progress as possible in one go.
In "advanced mode", one can list tens or hundreds of stages(repos) in the "2D list" of the API, to run in sequence. But when one stage fails for whatever reason, the whole pipeline halts, missing the opportunity to run the following stages and just wasting time for human intervention.
Better still, a timeout (say 4 hours) setting can also be useful, for those stages that don't actually fail but still are not making any progress due to rate limiting etc.
### Use case
Add a global option "skip stage when stalled", and a sub-setting for a timeout.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] mintsweet commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
mintsweet commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1302960999
@yumengwang03 what does this retry-like button do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
Startrekzky commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1281003183
> > > @asterix314 Thanks for the feedback, it makes sense to me, @Startrekzky what do you think?
> >
> >
> > I'm not sure, may I know what specific problem happened while you were collecting data from multiple GitHub repos? Thanks. @asterix314
>
> @Startrekzky I used the "2D list" API to collect a few hundred GitHub projects into DevLake. The stages ran in sequence. But when one stage fails, the whole pipeline stalls. There are many reasons for one stage to fail/stall:
>
> * miss-spelt project names
> * rate limit exceeded
> * etc.
>
> As a result, during the few days of the pipeline's execution, I found myself having to repeatedly return to the config-ui page to check if any stage failed and resume the pipeline manually. It would be much more efficient to automatically skip failed stages, and let the user intervene only at the end, when no executable stages are left unprocessed.
Thanks for the use case, I'll think about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] yumengwang03 commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
yumengwang03 commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1302907523
Figma link: https://www.figma.com/file/Eux4Dhorxp7M0TwzOnL6Yh/Config-UI-for-V1.0?node-id=2610%3A21361
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] mintsweet commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
mintsweet commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1305308534
@Startrekzky if I set up no skip failed task, can the task be rerun? (Does the rerun button show?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
Startrekzky commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1303067424
@mintsweet To rerun the subtask @mindlesscloud
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] github-actions[bot] commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1214483068
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
Startrekzky commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1305714256
> @Startrekzky if I set up no skip failed task, can the task be rerun? (Does the rerun button show?)
I think it's a `No`. What do you think? @mindlesscloud @klesh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] github-actions[bot] commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1280108853
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] asterix314 commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
asterix314 commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1219150039
> > @asterix314 Thanks for the feedback, it makes sense to me, @Startrekzky what do you think?
>
> I'm not sure, may I know what specific problem happened while you were collecting data from multiple GitHub repos? Thanks. @asterix314
I used the "2D list" API to collect a few hundred GitHub projects into DevLake. The stages ran in sequence. But when one stage fails, the whole pipeline stalls. There are many reasons for one stage to fail/stall:
- miss-spelt project names
- rate limit exceeded
- etc.
As a result, during the few days of the pipeline's execution, I found myself having to repeatedly return to the config-ui page to check if any stage failed and resume the pipeline manually. It would be much more efficient to automatically skip failed stages, and let the user intervene only at the end, when no executable stages are left unprocessed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
klesh commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1296484429
1. allow users to skip failed tasks
2. allow users to rerun failed/succeeded tasks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
klesh commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1185072512
@asterix314 Thanks for the feedback, it makes sense to me, @Startrekzky what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
Startrekzky commented on issue #2499:
URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1214537438
> @asterix314 Thanks for the feedback, it makes sense to me, @Startrekzky what do you think?
I'm not sure, may I know what specific problem happened while you were collecting data from multiple GitHub repos? Thanks. @asterix314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] likyh closed issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
Posted by GitBox <gi...@apache.org>.
likyh closed issue #2499: [Feature][Config UI] Skipping Failed/Stalled Stages
URL: https://github.com/apache/incubator-devlake/issues/2499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org