You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "AnkeshThakur (via GitHub)" <gi...@apache.org> on 2025/12/17 07:21:43 UTC
[D] What is the meaning of setting custom value for Time Range in "Set Sync Policy" [incubator-devlake]
GitHub user AnkeshThakur created a discussion: What is the meaning of setting custom value for Time Range in "Set Sync Policy"
I want to understand how does setting Time Range value impacts overall data collection and analysis. Specifically,
- Does it set the beginning date for data collection in each sync cycle? In other word it sets the look-back period's initial date in each sync
- Or is it just for the first time sync?
- Would it capture the data if I set this period to be 3 months and PRs which are older than 3 months are now getting merged?
<img width="670" height="207" alt="Screenshot 2025-12-17 at 12 32 22 PM" src="https://github.com/user-attachments/assets/3d7c0c86-0c87-43d4-9cf0-fc83ca25aad4" />
GitHub link: https://github.com/apache/incubator-devlake/discussions/8663
----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org
Re: [D] What is the meaning of setting custom value for Time Range in "Set Sync Policy" [incubator-devlake]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
GitHub user dosubot[bot] edited a comment on the discussion: What is the meaning of setting custom value for Time Range in "Set Sync Policy"
<!-- Answer -->
Setting a custom Time Range in "Set Sync Policy" defines the earliest date from which DevLake will collect data during syncs. This Time Range acts as the starting point for the initial historical data collection window. After the first successful sync, DevLake switches to incremental syncs, using the last successful sync time as the new lower bound for future data collection, not the original Time Range start date. So, the Time Range is mainly relevant for the first sync or if you reset the sync policy to an earlier date [source](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/collector_state_manager.go#L29-L119).
For most data sources (like PRs), the Time Range filter is applied to the creation time. This means if you set the period to 3 months, PRs created more than 3 months ago but merged now will not be collected by the main collector, because their creation date is outside the window [source](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/azuredevops_go/tasks/pr_collector.go#L65-L74).
However, DevLake has a secondary mechanism: if incremental sync is enabled and the plugin supports it, there is a secondary collector that can pick up updates to "unfinished" or recently updated records (like open PRs), even if they were created before the Time Range. This helps capture late merges or updates, but only for records that are still open or have recent activity [source](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/api_collector_stateful.go#L30-L269).
Here's a simplified diagram of the sync logic:
```mermaid
graph TD
A["Set Time Range (Start Date)"] --> B["First Sync: Collect data created after Start Date"]
B --> C["Incremental Syncs: Use last successful sync time as new lower bound"]
C --> D["Secondary Collector: Picks up updates to old records if supported"]
```
If you want to ensure you capture late updates to old PRs, consider setting a wider initial Time Range or check if your plugin supports the secondary collector for unfinished/updated records.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=reply-with-mention&utm_term=apache%2Fincubator-devlake).*
---
Docs are dead. Just use [Dosu](https://dosu.dev?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-tagline&utm_term=apache%2Fincubator-devlake).
[](https://app.dosu.dev/response-feedback?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-feedback&utm_term=apache%2Fincubator-devlake#message_id=8a10e72f-d666-48d9-86c4-55f58123fbed) [](https://github.dosu.com/apache/incubator-devlake?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-ask-repo&utm_term=apache%2Fincubator-devlake) [](https://app.dosu.dev/signup?referrer=openSource&source=github-footer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-share-team&utm_term=apache%2Fincubator-devlake)
GitHub link: https://github.com/apache/incubator-devlake/discussions/8663#discussioncomment-15276175
----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org
Re: [D] What is the meaning of setting custom value for Time Range in "Set Sync Policy" [incubator-devlake]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
GitHub user dosubot[bot] added a comment to the discussion: What is the meaning of setting custom value for Time Range in "Set Sync Policy"
<!-- Answer -->
Setting a custom Time Range in "Set Sync Policy" defines the earliest date from which DevLake will collect data during syncs. This Time Range acts as the starting point for the initial historical data collection window. After the first successful sync, DevLake switches to incremental syncs, using the last successful sync time as the new lower bound for future data collection, not the original Time Range start date. So, the Time Range is mainly relevant for the first sync or if you reset the sync policy to an earlier date [source](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/collector_state_manager.go#L29-L119).
For most data sources (like PRs), the Time Range filter is applied to the creation time. This means if you set the period to 3 months, PRs created more than 3 months ago but merged now will not be collected by the main collector, because their creation date is outside the window [source](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/azuredevops_go/tasks/pr_collector.go#L65-L74).
However, DevLake has a secondary mechanism: if incremental sync is enabled and the plugin supports it, there is a secondary collector that can pick up updates to "unfinished" or recently updated records (like open PRs), even if they were created before the Time Range. This helps capture late merges or updates, but only for records that are still open or have recent activity [source](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/api_collector_stateful.go#L30-L269).
Here's a simplified diagram of the sync logic:
```mermaid
graph TD
A["Set Time Range (Start Date)"] --> B["First Sync: Collect data created after Start Date"]
B --> C["Incremental Syncs: Use last successful sync time as new lower bound"]
C --> D["Secondary Collector: Picks up updates to old records if supported"]
```
If you want to ensure you capture late updates to old PRs, consider setting a wider initial Time Range or check if your plugin supports the secondary collector for unfinished/updated records.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/8a10e72f-d666-48d9-86c4-55f58123fbed?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/8a10e72f-d666-48d9-86c4-55f58123fbed?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/8a10e72f-d666-48d9-86c4-55f58123fbed?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/8a10e72f-d666-48d9-86c4-55f58123fbed?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/8a10e72f-d666-48d9-86c4-55f58123fbed?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/8a10e72f-d666-48d9-86c4-55f58123fbed?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/8a10e72f-d666-48d9-86c4-55f58123fbed?feedback_type=other)</sup>
[](https://app.dosu.dev/b4e8e847-d479-4541-83a8-d88d83fea5c9/ask?utm_source=github) [](https://cursor.com/link/prompt?text=Setting%20a%20custom%20Time%20Range%20in%20%22Set%20Sync%20Policy%22%20defines%20the%20earliest%20date%20from%20which%20DevLake%20will%20collect%20data%20during%20syncs.%20This%20Time%20Range%20acts%20as%20the%20starting%20point%20for%20the%20initial%20historical%20data%20collection%20window.%20After%20the%20first%20successful%20sync%2C%20DevLake%20switches%20to%20incremental%20syncs%2C%20using%20the%20last%20successful%20sync%20time%20as%20the%20new%20lower%20bound%20for%20future%20data%20collection%2C%20not%20the%20original%20Time%20Range%20start%20date.%20So%2C%20the%20Time%20Range%20is%20mainly%20relevant%20for%20the%20first%20sync%20or%20if%20you%20reset%20the%20sync%20policy%20to%20an%20earlier%20date%20%5Bsource%5D%28https%3A//github.com/apache
/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/collector_state_manager.go%23L29-L119%29.%0A%0AFor%20most%20data%20sources%20%28like%20PRs%29%2C%20the%20Time%20Range%20filter%20is%20applied%20to%20the%20creation%20time.%20This%20means%20if%20you%20set%20the%20period%20to%203%20months%2C%20PRs%20created%20more%20than%203%20months%20ago%20but%20merged%20now%20will%20not%20be%20collected%20by%20the%20main%20collector%2C%20because%20their%20creation%20date%20is%20outside%20the%20window%20%5Bsource%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/azuredevops_go/tasks/pr_collector.go%23L65-L74%29.%0A%0AHowever%2C%20DevLake%20has%20a%20secondary%20mechanism%3A%20if%20incremental%20sync%20is%20enabled%20and%20the%20plugin%20supports%20it%2C%20there%20is%20a%20secondary%20collector%20that%20can%20pick%20up%20updates%20to%20%22unfinished%22%20or%20recently%20updated%20records%20%2
8like%20open%20PRs%29%2C%20even%20if%20they%20were%20created%20before%20the%20Time%20Range.%20This%20helps%20capture%20late%20merges%20or%20updates%2C%20but%20only%20for%20records%20that%20are%20still%20open%20or%20have%20recent%20activity%20%5Bsource%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/api_collector_stateful.go%23L30-L269%29.%0A%0AHere%27s%20a%20simplified%20diagram%20of%20the%20sync%20logic%3A%0A%0A%60%60%60mermaid%0Agraph%20TD%0A%20%20%20%20A%5B%22Set%20Time%20Range%20%28Start%20Date%29%22%5D%20--%3E%20B%5B%22First%20Sync%3A%20Collect%20data%20created%20after%20Start%20Date%22%5D%0A%20%20%20%20B%20--%3E%20C%5B%22Incremental%20Syncs%3A%20Use%20last%20successful%20sync%20time%20as%20new%20lower%20bound%22%5D%0A%20%20%20%20C%20--%3E%20D%5B%22Secondary%20Collector%3A%20Picks%20up%20updates%20to%20old%20records%20if%20supported%22%5D%0A%60%60%60%0A%0AIf%20you%20want%20to%20ensure%20you%20capt
ure%20late%20updates%20to%20old%20PRs%2C%20consider%20setting%20a%20wider%20initial%20Time%20Range%20or%20check%20if%20your%20plugin%20supports%20the%20secondary%20collector%20for%20unfinished/updated%20records.) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/incubator-devlake/discussions/8663)
GitHub link: https://github.com/apache/incubator-devlake/discussions/8663#discussioncomment-15276175
----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org
Re: [D] What is the meaning of setting custom value for Time Range in "Set Sync Policy" [incubator-devlake]
Posted by "AnkeshThakur (via GitHub)" <gi...@apache.org>.
GitHub user AnkeshThakur edited a discussion: What is the meaning of setting custom value for Time Range in "Set Sync Policy"
I want to understand how does setting Time Range value impact overall data collection and analysis. Specifically,
- Does it set the beginning date for data collection in each sync cycle? In other word it sets the look-back period's initial date in each sync
- Or is it just for the first time sync?
- Would it capture the data if I set this period to be 3 months and PRs which are older than 3 months are now getting merged?
<img width="670" height="207" alt="Screenshot 2025-12-17 at 12 32 22 PM" src="https://github.com/user-attachments/assets/3d7c0c86-0c87-43d4-9cf0-fc83ca25aad4" />
GitHub link: https://github.com/apache/incubator-devlake/discussions/8663
----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org