You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "benjaminrmoss (via GitHub)" <gi...@apache.org> on 2024/07/25 14:52:39 UTC
[I] [Question][AzureDevOps] Pipeline run retention [incubator-devlake]
benjaminrmoss opened a new issue, #7788:
URL: https://github.com/apache/incubator-devlake/issues/7788
Hi - The ADO project I'm working with has a pipeline rention policy of 3 previous runs. When running the data collection in DevLake, any runs that have expired or been deleted from ADO are removed, which leads to the DORA dashboard having no/less data.
Is there any way for me to configure DevLake to retain any pipeline metrics even if the actual pipeline run in ADO has been removed? Thanks for your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][AzureDevOps] Pipeline run retention [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7788:
URL: https://github.com/apache/incubator-devlake/issues/7788#issuecomment-2381035510
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][AzureDevOps] Pipeline run retention [incubator-devlake]
Posted by "d4x1 (via GitHub)" <gi...@apache.org>.
d4x1 commented on issue #7788:
URL: https://github.com/apache/incubator-devlake/issues/7788#issuecomment-2258067947
@benjaminrmoss You can try azuredevop_go, it can collect data incremantally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][AzureDevOps] Pipeline run retention [incubator-devlake]
Posted by "d4x1 (via GitHub)" <gi...@apache.org>.
d4x1 commented on issue #7788:
URL: https://github.com/apache/incubator-devlake/issues/7788#issuecomment-2252864670
@benjaminrmoss You can run your project in `incremental` mode. It will only collect data that changes therotically since last successful sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][AzureDevOps] Pipeline run retention [incubator-devlake]
Posted by "benjaminrmoss (via GitHub)" <gi...@apache.org>.
benjaminrmoss commented on issue #7788:
URL: https://github.com/apache/incubator-devlake/issues/7788#issuecomment-2253001368
@d4x1 Thanks for the reply. Would I set this in the project configuration settings? I can't see any option for it here:

Maybe this feature isn't supported for ADO yet? https://devlake.apache.org/docs/Overview/SupportedDataSources/#azure-devops
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][AzureDevOps] Pipeline run retention [incubator-devlake]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #7788:
URL: https://github.com/apache/incubator-devlake/issues/7788#issuecomment-2250596491
<!-- Greeting -->
Hey @benjaminrmoss! I'm a bot here to help you out with bugs, questions, and more. Let's tackle this issue together!
<!-- Answer -->
Yes, DevLake allows for the retention of pipeline run data independently of the source system's retention policies through its synchronization policy settings. You can configure the sync frequency, time range, and the skip-on-fail option for your data, which helps in managing how and when data is collected and retained <sup>[[1]](https://github.com/apache/incubator-devlake-website/blob/main/versioned_docs/version-v0.21/Configuration/Tutorial.md)</sup><sup>[[2]](https://github.com/apache/incubator-devlake-website/blob/main/versioned_docs/version-v1.0/Configuration/Tutorial.md)</sup>.
Additionally, DevLake supports different data collection modes, including "Collect Data (Default)," "Collect Data in Full Refresh Mode," and "Re-transform Data," which provide flexibility in how data is managed and retained <sup>[[1]](https://github.com/apache/incubator-devlake-website/blob/main/versioned_docs/version-v0.21/Configuration/Tutorial.md)</sup><sup>[[2]](https://github.com/apache/incubator-devlake-website/blob/main/versioned_docs/version-v1.0/Configuration/Tutorial.md)</sup>.
For ensuring that historical pipeline run data is preserved, you can follow these best practices:
1. **Set Up a Blueprint with a Regular Sync Schedule**: Configure your blueprint for Azure DevOps data collection with a regular sync schedule to automatically trigger historical runs at specified intervals.
2. **Manual Execution of Historical Runs**: Manually trigger historical runs via the Configuration UI or API for ad-hoc data collection or backfilling data.
3. **Monitor and Manage Historical Runs**: Regularly monitor historical runs to ensure successful execution and address any errors promptly.
4. **Use the API for Advanced Control**: Utilize the DevLake API for more granular control over data collection and transformation processes <sup>[[3]](https://github.com/apache/incubator-devlake-website/blob/main/versioned_docs/version-v0.17/Overview/KeyConcepts.md)</sup>.
By following these practices, you can ensure that historical pipeline run data is consistently preserved in DevLake, even if the actual pipeline runs in Azure DevOps have been removed.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org