[D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["vasukiiyengar98 (via GitHub)" <gi...@apache.org>]

GitHub user vasukiiyengar98 created a discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

Hi DevLake team,

We are using Apache DevLake v1.0.2 with the GitHub plugin to collect GitHub Actions workflow runs.

Our DevLake deployment is running on AWS ECS. We have created the required ECS containers/services using the official Apache DevLake Docker images.

We have some GitHub Actions workflows that run for several hours. During the "Collect Workflow Runs" task, DevLake logs messages like:

Skipping run{id: 3372***, number: 1158***} with status in_progress
Skipping run{id: 3372***, number: 303***} with status in_progress

Our concern is that these workflow runs are skipped while they are still in_progress.

Later, when the GitHub Actions workflow finishes and its status changes to completed, a subsequent incremental sync does not appear to collect or update the previously skipped workflow run.

Our expected behavior is:

1. Detect the workflow run while it is in_progress.
2. Keep it as an unfinished/in-progress run.
3. Re-check the run during subsequent syncs.
4. Once GitHub reports the run as completed, update the existing workflow run record.

Could you please confirm:

- Is this expected behavior in DevLake v1.0.2?
- Is there any configuration that makes DevLake re-check in_progress workflow runs during later syncs?
- Has this behavior been fixed or changed in a newer DevLake version?
- If not, what is the recommended approach for handling long-running GitHub Actions workflows?

Since our DevLake deployment is running on AWS ECS using the official DevLake Docker images, we would prefer to use a supported configuration or upgrade rather than maintain a custom DevLake code/image if possible.

Any guidance on the recommended approach would be greatly appreciated.

Thanks!

GitHub link: https://github.com/apache/devlake/discussions/9125

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org

Re: [D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["NhanDuong21 (via GitHub)" <gi...@apache.org>]

GitHub user NhanDuong21 added a comment to the discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

There is now an upstream fix for this, newer than the replies above: [#9162](https://github.com/apache/devlake/pull/9162) was merged on September 24 and is included in [v1.0.3-beta18](https://github.com/apache/devlake/releases/tag/v1.0.3-beta18).

The collector now stores unfinished runs and revisits the oldest unfinished run when choosing the next incremental collection window. That lookback is bounded to 14 days, so your 3–4-hour workflows fall within it. This does not require a custom image.

I would back up the database, test the beta18 release images in a staging copy of your ECS deployment, and check one known run through both `in_progress` and `completed` using your existing two-hour incremental schedule. The release's Compose attachment lists the matching backend, Config UI and dashboard image tags.

One important distinction: runs discarded by v1.0.2 are not automatically recovered by the new unfinished-run tracking. The fix documents a one-time backfill with `TimeAfter` covering their creation dates. Since your previous full refresh removed dashboard history, do not repeat a one-month refresh on production without a backup and a staging test covering the full history you intend to retain. Data that GitHub no longer exposes may require restoring your earlier backup.

GitHub link: https://github.com/apache/devlake/discussions/9125#discussioncomment-18720878

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org

Re: [D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["vasukiiyengar98 (via GitHub)" <gi...@apache.org>]

GitHub user vasukiiyengar98 added a comment to the discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

Hi @klesh, thanks for your reply!

I tried the Full Refresh approach suggested by @tomm1990, but unfortunately, after the refresh I lost some of the data that was previously available.

If this issue with in_progress workflow runs has been fixed or improved in any newer version, upgrading would be very useful for us, since we need to retain and track all workflow-run data from the past few months. Skipping these runs doesn’t seem like a reliable solution.

I’m also not sure how others have worked around this issue. If there’s anything else I can try from my side, please let me know.

Thanks again for your help!

GitHub link: https://github.com/apache/devlake/discussions/9125#discussioncomment-18443816

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org

Re: [D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["klesh (via GitHub)" <gi...@apache.org>]

GitHub user klesh added a comment to the discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

I don't recall this issue was fixed in newer version.

GitHub link: https://github.com/apache/devlake/discussions/9125#discussioncomment-18401828

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org

Re: [D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["vasukiiyengar98 (via GitHub)" <gi...@apache.org>]

GitHub user vasukiiyengar98 added a comment to the discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

Thanks for the detailed explanation.

Before we proceed with the full/backfill sync approach, could you please confirm if there is any update, fix, configuration, or recommended workaround available in any newer DevLake version or branch that addresses this behavior?

We are currently using v1.0.2 and would prefer to use an official DevLake release/configuration rather than maintain a custom image.

If there is no official solution available yet, could you also let us know if there is an existing issue, PR, or planned change for supporting in_progress workflow runs and updating them once they become completed?

Thanks for your guidance.

GitHub link: https://github.com/apache/devlake/discussions/9125#discussioncomment-18352303

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org

Re: [D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["tomm1990 (via GitHub)" <gi...@apache.org>]

GitHub user tomm1990 added a comment to the discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

In `backend/plugins/github/tasks/cicd_run_collector.go` (tag `v1.0.2`) Collect Workflow Runs only keeps runs with `status == "completed"`. Anything else (including `in_progress`) is logged as `Skipping run{id: ..., number: ...} with status ...` and discarded. Nothing is written for those IDs, so a later sync cannot “update the existing unfinished record.”

that skip is expected in v1.0.2, and there is no toggle to “keep in_progress and update later.”

Worse for multi-hour Actions: that collector is finalizable on `created_at`. Incremental sync advances `since` from collected completed runs. A long run that was skipped while in progress still has the old `created_at`. Once `since` moves past that timestamp, a later completed sync may never walk back far enough to see it again. That matches what you are seeing.

Current `main` still filters to completed only (it got quieter about the skip log and uses `created=from..to` windows). So upgrading the official image alone does not give you “store in_progress then patch on completion.”

Practical options without a custom image:

1. After long workflows finish (or on a schedule), run a **full sync** (or a collect with `timeAfter` / time range that covers those runs’ `created_at`), so the completed rows are listed again.
2. Keep incremental for day-to-day, and treat full/backfill sync as the safety net for hour-scale jobs.
3. If you need true in-progress tracking, that is a product change (persist non-completed + refresh by id/`updated_at`). Worth an issue against `apache/devlake` with your ECS image tag and the skip log lines.

If you can paste one skipped run id and whether a full sync afterward creates the `_tool_github_runs` row, that confirms the created_at window miss vs a different filter problem.

GitHub link: https://github.com/apache/devlake/discussions/9125#discussioncomment-18351581

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org

Re: [D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["tomm1990 (via GitHub)" <gi...@apache.org>]

GitHub user tomm1990 added a comment to the discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

@vasukiiyengar98 you are right that there is no Config UI toggle for a *recurring* full/backfill sync. The cron you set (`0 */2 * * *`) always runs **Collect Data (Default)** = incremental. that is why a 2h schedule alone cannot recover a 3-4h run that was skipped while `in_progress`.

here is how to do the workaround on stock **v1.0.2** (Config UI + API):

### 1. Time Filter (`timeAfter`) - blueprint Sync Policy

In Config UI: open the **project / blueprint** → **Sync Policy** (or edit policy).

- **Time Filter** is the blueprint `timeAfter` field (docs call it Time Filter).
- set it far enough back that a full refresh can still see those runs’ `created_at` (for 3-4h jobs, use at least several days, e.g. last 7 days / last month - not “today only”).

that bound applies to both default collect and full refresh. it does **not** by itself make incremental re-walk past the collector’s stored `since`.

### 2. One-shot full refresh from the UI

On the same blueprint detail page, use the collect control (not the cron):

- **Collect Data** = incremental (same as your schedule)
- **Collect Data in Full Refresh Mode** = delete/re-collect inside the Time Filter window (`fullSync: true`)

run Full Refresh Mode after a long workflow finishes (or on a schedule you own). that is the UI path that re-lists completed Actions runs whose `created_at` would otherwise sit behind incremental `since`.

### 3. Recurring full refresh via API (recommended on ECS)

Config UI has no “full sync every N hours” cron. trigger it yourself against the lake API:

```bash
curl -X POST "http://<devlake-api-host>/blueprints/<blueprintId>/trigger" \
  -H "Content-Type: application/json" \
  -d '{"skipCollectors": false, "fullSync": true}'
```

wire that to EventBridge / a small ECS scheduled task (e.g. every 6h or daily). keep your existing 2h blueprint cron for normal incremental, and use this as the safety net for hour-scale Actions.

`blueprintId` is in the Config UI URL / blueprint detail; API host is your DevLake API service (same network as Config UI).

### 4. Future version / planned fix

there is still **no** public release or roadmap item I can point to that will store `in_progress` and update on completion. I cannot honestly name a future version for that. filing an issue on `apache/devlake` (GitHub plugin, image tag `v1.0.2`, skip log lines, one run id missed by incremental but present after full refresh) is the path to get it on the official backlog.

practical combo for your 3-4h prod workflows: keep `0 */2 * * *` + set Time Filter ≥ several days + schedule `fullSync: true` triggers after those jobs (or on a fixed cadence).

GitHub link: https://github.com/apache/devlake/discussions/9125#discussioncomment-18366719

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org

Re: [D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["vasukiiyengar98 (via GitHub)" <gi...@apache.org>]

GitHub user vasukiiyengar98 added a comment to the discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

Hi @tomm1990, I tested the Full Refresh approach with the Time Filter set to 1 month. manually running the workflow

Our current DevLake version is v1.0.2.

The Full Refresh completed successfully, but after the refresh I noticed that a significant amount of data that was previously visible in our Grafana dashboard is now missing. I compared the dashboard before and after the Full Refresh, and the difference is quite noticeable.

I also checked the GitHub collector logs, and the Full Refresh is still skipping multiple workflow runs with in_progress status.

So it looks like the Full Refresh is re-collecting the data within the Time Filter, but it is still skipping in_progress runs, and it also doesn't seem to restore all of the data we previously had.

Could you please help me understand what might be happening here? Is there anything else I should check in the Time Filter or Full Refresh configuration before we proceed further?

Also, since we are currently using v1.0.2, is there any newer DevLake version that has resolved or improved this type of GitHub Actions workflow-run issue? If so, could you please let me know which version you recommend? I can test the newer version in our environment before going further with the workaround.

Thanks again for your help and for taking the time to look into this.

GitHub link: https://github.com/apache/devlake/discussions/9125#discussioncomment-18368500

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org

Re: [D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["vasukiiyengar98 (via GitHub)" <gi...@apache.org>]

GitHub user vasukiiyengar98 added a comment to the discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

Thanks a lot for checking this and for the detailed explanation.

In our setup, we have configured the GitHub Workflow Runs sync policy to run every 2 hours (0 */2 * * *), because we have production workflows that can run for around 3–4 hours.

Our intention is to capture those workflow runs through the regular incremental sync once they complete. However, because the runs are skipped while they are in_progress, the later incremental sync doesn't pick them up after they become completed.

You mentioned that a full/backfill sync or a timeAfter window covering the workflow runs' created_at can be used as a workaround. Could you please explain how we can configure this in our current DevLake version (v1.0.2)?

We currently don't see an option in the UI to configure a recurring full/backfill sync or a timeAfter window for this purpose. If there is a specific configuration, API, or task setting we should use, could you please point us in the right direction?

Also, if this behavior is something planned to be fixed in the GitHub plugin, it would be very helpful to know which future version we could expect the fix in.

Thanks again for your help. DevLake is a great platform and it has already helped us solve several issues with our engineering data and reporting. We really appreciate the work from the team and community.

GitHub link: https://github.com/apache/devlake/discussions/9125#discussioncomment-18366155

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org

Re: [D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["vasukiiyengar98 (via GitHub)" <gi...@apache.org>]

GitHub user vasukiiyengar98 added a comment to the discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

@tomm1990 , Thanks a lot for your detailed explanation and for taking the time to look into this. I really appreciate your help.
I’ll go through the steps and test the workaround on our setup. If I face any issues or have any questions while testing, I’ll update them here. Thanks again for your support!

GitHub link: https://github.com/apache/devlake/discussions/9125#discussioncomment-18367080

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org

Re: [D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["tomm1990 (via GitHub)" <gi...@apache.org>]

GitHub user tomm1990 added a comment to the discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

@vasukiiyengar98 short answer: no official release or config I can find that keeps `in_progress` runs and updates them when they complete. Full/backfill sync is still the supported workaround on stock images.

I checked beyond v1.0.2:

- `v1.0.3` / `v1.0.4-alpha`: same completed-only filter + `Skipping run ... with status ...` path as 1.0.2
- `main` / recent `v1.0.3-beta*`: collector was rewritten for the `/actions/runs` pagination caps ([#8844](https://github.com/apache/devlake/pull/8844) / [#8842](https://github.com/apache/devlake/issues/8842)), but it still only keeps `status == "completed"` (comment in code: “only keep completed runs”). No in_progress persist + refresh

So upgrading the official image can help large-repo collection reliability. It does not change the long-running Actions miss you hit.

I also could not find an existing issue/PR specifically for “store in_progress GitHub Actions runs and update on completion.” Closest related work is the pagination fix above, which is a different problem. If you need that behavior, filing a new GitHub issue against `apache/devlake` (plugin=GitHub, your ECS image tag, skip log lines, and one run id that never lands after incremental) is the right next step for an official fix.

Until then: stay on official images, keep incremental for day-to-day, and schedule full sync / a `timeAfter` window that covers those runs’ `created_at` after long workflows finish.

GitHub link: https://github.com/apache/devlake/discussions/9125#discussioncomment-18364053

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org

Re: [D] [GitHub] Workflow runs with in_progress status are skipped and not collected after completion [devlake]

["tomm1990 (via GitHub)" <gi...@apache.org>]

GitHub user tomm1990 added a comment to the discussion: [GitHub] Workflow runs with in_progress status are skipped and not collected after completion

@vasukiiyengar98 sounds good - once Full Refresh Mode / the `fullSync: true` trigger picks up a previously skipped run, please mark an answer on this thread so the next ECS + long Actions setup finds it. if something still misses after that, paste the run id + whether `_tool_github_runs` got the row and we can dig from there.

GitHub link: https://github.com/apache/devlake/discussions/9125#discussioncomment-18368403

----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org