You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "rodrigoluizs (via GitHub)" <gi...@apache.org> on 2025/04/11 15:15:38 UTC
[I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
rodrigoluizs opened a new issue, #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
## Question
Hi folks,
We’re using DevLake to track DORA metrics, and we’ve noticed that automated commits and PRs (e.g., from Renovate or Dependabot) are skewing our metrics — especially lead time and deployment frequency. These automated changes usually have close to zero code time and very low pickup/review time, since engineers often merge them as soon as the test pipelines pass. In some cases, they are even automatically merged when all the pipelines succeed.
**Is there already a way in DevLake to exclude commits or PRs from specific authors (like bots) when calculating DORA metrics**? If not, we’d be interested in contributing a solution, as we believe this could be useful for others as well.
Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-2889545240
Hi @rodrigoluizs ,
1. For the filtering method, I don't have a preference between the two. And I have two further questions regarding the option 2:
- Which tables will the `is_bot` flag be added, only table `accounts`?
- Which cases will the `is_bot` take effect? I'm not worrying about the dashboard queries would be the plugin's internal queries, for instance, the calculation in the DORA plugins to generate table `project_pr_metrics` might also take the `bot` PRs or commits. Thus, it should be updated after the `is_bot` flag is introduced.
2. For bot detection, we could use the `enviroment variables` with default values to achieve both `auto + manual control`
3. For Grafana dashboards, if the `is_bot` is added, updating the SQL in the existing dashboard would be my choice.
Cc. @klesh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-3568543726
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "rodrigoluizs (via GitHub)" <gi...@apache.org>.
rodrigoluizs commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-2842399237
Thanks @klesh !
I went ahead and did a bit of analysis on how this could be implemented.
Below is a draft proposal with possible approaches, pros/cons, and a few open questions for feedback. Let me know what you think — happy to iterate on it or prepare an RFC based on where you’d see the most value!
# Excluding Automated Bot Changes from DORA Metrics
---
## 1. Filtering Approaches
I'm thinking about two main approaches to apply bot filtering in the **queries used by Grafana dashboards**:
### Option 1: Filter dynamically by author name pattern (e.g., `%bot%`)
**SQL example:**
```sql
SELECT a.full_name, prm.*
FROM devlake.project_pr_metrics prm
INNER JOIN devlake.pull_requests pr ON prm.id = pr.id
INNER JOIN devlake.accounts a ON pr.author_name = a.full_name
WHERE a.full_name NOT LIKE '%bot%'
```
**Pros:**
- No schema changes needed
- Quick to implement
**Cons:**
- Pattern matching is error-prone (some bots might not match)
- Makes queries heavier and harder to maintain
- Hard to manually override for special cases
---
### Option 2: Introduce a flag to pre-classify bots and filter on it
Add a boolean `is_bot` field to the `accounts` table and propagate it into `project_pr_metrics` with a new `is_authored_by_bot` field.
**SQL example:**
```sql
SELECT *
FROM devlake.project_pr_metrics
WHERE NOT is_authored_by_bot
```
**Pros:**
- Queries stay clean and fast
- Easier to manually control (allow exceptions, etc.)
- More future-proof if bot naming conventions change
**Cons:**
- Requires schema changes and changes to data ingestion
---
## 2. How to Populate the New Flag
In case we decide to go for a new flag, we see two options for how the `is_bot` flag could be populated:
### Option 1: Automatic Detection
- When saving accounts, automatically set `is_bot = true` if:
- `full_name` or `user_name` contains typical bot patterns like `bot`, `renovate`, `dependabot`, etc.
**Pros:**
- No manual maintenance needed
- Works for most typical bots
**Cons:**
- Might incorrectly miss or misclassify accounts if naming conventions vary
- This could be addressed by allowing the bot name patterns to be defined in the Git integration configs
---
### Option 2: Manual Control
- Provide an API endpoint (or reuse an existing admin tool) that allows us to manually set or update `is_bot` on accounts.
**Pros:**
- Full flexibility for maintainers
- Correct classification possible even for unusual bot names
**Cons:**
- Requires manual effort to maintain the bot list
---
**Note:**
Both approaches could potentially be combined if preferred — automatic detection for most accounts, with manual overrides where needed.
---
## 3. Grafana Dashboard Behavior
We also considered two options for displaying the filtered DORA metrics:
### Option 1: Add a new variable to existing dashboards
- New Grafana variable: `include_bots`
- Values: `true`, `false`
- Default: `true` (bots included) to preserve current behavior and avoid surprising users
- Queries would check the variable:
```sql
WHERE (${include_bots} = 'true' OR is_authored_by_bot = FALSE)
```
**Pros:**
- No need to maintain separate dashboards
- Easy for users to switch views
- Low maintenance overhead
**Cons:**
- Slightly more complex queries
- Might be overlooked without proper UI hint
---
### Option 2: Create separate dashboards excluding bots
- Clone existing DORA dashboards
- Hardcode filtering (`WHERE is_authored_by_bot = FALSE`)
- Label clearly, e.g., “DORA Metrics (Human-only Changes)”
**Pros:**
- Very clear and easy for users to find
- No need for users to toggle anything
**Cons:**
- Need to maintain two sets of dashboards
- Possible risk of dashboard drift over time
---
## Summary
| Decision Point | Option 1 | Option 2 |
|----------------------|---------------------------------------------------|----------------------------------------------------|
| Filtering Method | Author name pattern matching | Pre-calculated `is_bot` flag |
| Bot Detection | Automatic detection | Manual control (or both combined) |
| Grafana Dashboards | Add a variable to existing dashboards | Create new dashboards for human-only metrics |
---
## Open Questions for Maintainers
- What’s the preferred approach for filtering out automated changes — would dynamic filtering (e.g., by name pattern), a flag-based approach, or something else work best within DevLake’s model?
- For identifying bots, do you see more value in automatic detection, manual, both combined or would you suggest a different mechanism?
- What would be the best way to surface filtered metrics in Grafana — a dashboard variable, a separate dashboard, or another method?
- If this sounds good, should I prepare a full RFC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-2803695228
Not that I know of.
The idea of it sound great to me, feel free to do so and let me know if you need any help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-3100099252
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-2797213464
<!-- Greeting -->
Hello @rodrigoluizs! I'm [Dosu](https://dosu.dev), here to help you with bugs, questions, or even becoming a contributor. While you're waiting for a human maintainer, feel free to ask anything!
<!-- Answer -->
I found a similar closed issue that might be relevant to your question. In the issue titled "[Feature][Dashboard] Robot removal," a solution was implemented to address the exclusion of bot-generated data. This issue was closed by a pull request that added bot removal patterns to the PR metric documentation <sup>[[1]](https://github.com/apache/incubator-devlake/issues/3339)</sup>.
If you're interested in contributing a solution to exclude commits or PRs from specific authors like bots, you can follow the feature request process outlined in the feature request template. This involves searching for existing issues, providing a detailed description of your proposed feature, and indicating your willingness to submit a pull request <sup>[[2]](https://github.com/apache/incubator-devlake/blob/main/.github/ISSUE_TEMPLATE/feature-request.yml)</sup>.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/c697331d-699d-47dc-b174-456525a432b8?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/c697331d-699d-47dc-b174-456525a432b8?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/c697331d-699d-47dc-b174-456525a432b8?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/c697331d-699d-47dc-b174-456525a432b8?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/c697331d-699d-47dc-b174-456525a432b8?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/c697331d-699d-47dc-b174-456525a432b8?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/c697331d-699d-47dc-b174-456525a432b8?feedback_type=other)</sup>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-2870775257
Apologies for the delayed response. Thanks for the well-written proposal; all options are clear and well-defined. I believe that the `is_bot` flag is a good approach and offers long-term flexibility. I suggest combining automated detection (initial identification) with manual overrides for optimal flexibility. Exposing the bot name pattern via an environment variable would provide further customization. Regarding the dashboards, I don't have a strong preference at the moment. Perhaps @Startrekzky can offer some insights?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "rodrigoluizs (via GitHub)" <gi...@apache.org>.
rodrigoluizs commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-3328181061
Implementation is still in progress. Please keep the issue open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-3594041738
This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-2899774341
Sounds solid to me.
I am fine with the `include_bots`, with or without it.
I agree with the column names.
> Additionally, I’d appreciate some clarification on how new environment variables can be introduced in DevLake, as I’m not very familiar with that part of the project yet.
Add it to the `env.example` file with a comment to describe the variable, then update https://github.com/apache/incubator-devlake/blob/main/backend/core/config/config_viper.go#L102 if you want to give it a default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "rodrigoluizs (via GitHub)" <gi...@apache.org>.
rodrigoluizs commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-3111861432
Implementation is still in progress. Please keep the issue open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-3325984026
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "rodrigoluizs (via GitHub)" <gi...@apache.org>.
rodrigoluizs commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-2898676920
Thanks for the feedback and questions, @Startrekzky and @klesh!
> **Which tables will the `is_bot` flag be added, only table accounts?**
Yes, the `is_bot` flag would be added only to the `accounts` table. In addition to that, my plan was to add a new column to `project_pr_metrics` called `is_authored_by_bot`, since I believe that name better reflects the context of the pull request entity and makes the intent clearer when querying.
> **Which cases will the `is_bot` take effect? I'm not worrying about the dashboard queries but the plugin's internal processing logic, for instance, the calculation in the DORA plugin to generate table `project_pr_metrics` might also take the bot PRs or commits. If so, the plugin's processing logic needs to be updated as well after the `is_bot` flag is introduced.**
You’re absolutely right — for this to work reliably, the DORA plugin’s processing logic that populates `project_pr_metrics` would also need to be updated to propagate the `is_bot` value from the author account into the `is_authored_by_bot` field during metric calculation. That way, downstream queries (like Grafana dashboards) can filter without needing to join back to the `accounts` table.
> **For bot detection, we could use the environment variables with default values to achieve both auto + manual control**
Using environment variables to control the bot name patterns sounds like a great way to support both automatic detection and manual overrides — I’ll incorporate that into the plan as well.
> **For Grafana dashboards, if the `is_bot` is added, updating the SQL in the existing dashboard would be my choice.**
That was also my preferred approach — nice to hear that you agree!
Just a small note: my intention was to filter on the new `is_authored_by_bot` column in `project_pr_metrics`.
---
### Follow-up
Based on your input, my current understanding of the preferred direction is:
1. **Filtering approach:** Use **Option 2** — introduce a flag (`is_bot` in `accounts`, and `is_authored_by_bot` in `project_pr_metrics`)
2. **Bot detection:** Combine **automatic detection** with **manual override**, using an environment variable to define bot name patterns
3. **Dashboard behavior:**
- Update existing dashboards to support filtering based on `is_authored_by_bot`
- I’d like your feedback on the idea to introduce an **`include_bots` variable** to control whether bot-authored changes should be filtered in the queries or not.
The idea here is to avoid introducing a breaking change and to keep the DORA metrics the same for users who do not explicitly opt in to this new feature.
---
Does this align with how you both see it?
Do you agree with the proposed column names — `is_bot` for the `accounts` table and `is_authored_by_bot` for `project_pr_metrics`?
Additionally, I’d appreciate some clarification on how new environment variables can be introduced in DevLake, as I’m not very familiar with that part of the project yet.
Just want to make sure we’re on the same page before moving forward with an RFC or implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #8381:
URL: https://github.com/apache/incubator-devlake/issues/8381#issuecomment-2899832107
@rodrigoluizs Looks great to me.
Based on this approach, you might need to touch every 'git related plugin (gitlab, github, bitbucket)' to populate the newly added fields. @klesh is our lead developer, and please let him know when you have any questions later, here or in Slack.
Thank you for the proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question][DORA] Ignoring Automated VCS Changes [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8381: [Question][DORA] Ignoring Automated VCS Changes
URL: https://github.com/apache/incubator-devlake/issues/8381
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org