This is an automated email from the ASF dual-hosted git repository.
zky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
The following commit(s) were added to refs/heads/main by this push:
new 2ec5065466 docs(website): add cicd_deployments table, update github/gitlab's entities (#671)
2ec5065466 is described below
commit 2ec50654668e8a3917fc77ba8c636f56178830a2
Author: Lynwee <li...@merico.dev>
AuthorDate: Tue Nov 7 18:52:53 2023 +0800
docs(website): add cicd_deployments table, update github/gitlab's entities (#671)
* docs(website): add cicd_deployments table, update github/gitlab's entities
* docs(website): update github's scope config screenshoot
* docs(website): fix some typos
* docs(website): fix typos and update cicd_deployment_commits table's comment
---
docs/Configuration/GitHub.md | 12 +--
.../images/github-set-transformation2.png | Bin 388738 -> 563178 bytes
.../images/gitlab-set-transformation2.png | Bin 181320 -> 225247 bytes
docs/DataModels/DevLakeDomainLayerSchema.md | 113 +++++++++++++--------
docs/Overview/SupportedDataSources.md | 108 ++++++++++----------
static/img/ConfigUI/gitlab-set-data-scope.png | Bin 76342 -> 239716 bytes
6 files changed, 132 insertions(+), 101 deletions(-)
diff --git a/docs/Configuration/GitHub.md b/docs/Configuration/GitHub.md
index cfe3b1f39a..d451050f25 100644
--- a/docs/Configuration/GitHub.md
+++ b/docs/Configuration/GitHub.md
@@ -19,7 +19,7 @@ Give your connection a unique name to help you identify it in the future.
#### Endpoint URL
-This should be a valid REST API endpoint, eg. `https://api.github.com/`. The URL should end with `/`.
+This should be a valid REST API endpoint, e.g. `https://api.github.com/`. The URL should end with `/`.
#### Personal Access Token(s)
@@ -92,14 +92,14 @@ Choose the GitHub repositories you wish to collect either by finding them in the
### Step 1.3 - Add Scope Config (Optional)
Scope config contains two parts:
-- The entities of which domain you wish to collect: Usually, you don't have to modify this part. However, if you don't want to collect certain GitHub entities, you can unselect some entities to accerlerate the collection speed.
+- The entities of which domain you wish to collect: Usually, you don't have to modify this part. However, if you don't want to collect certain GitHub entities, you can unselect some entities to accelerate the collection speed.
- Issue Tracking: GitHub issues, issue comments, issue labels, etc.
- Source Code Management: GitHub repos, refs, commits, etc.
- Code Review: GitHub PRs, PR comments and reviews, etc.
- CI/CD: GitHub Workflow runs, GitHub Workflow jobs, etc.
- Cross Domain: GitHub accounts, etc.
- The transformations on the GitHub data you are going to collect.
- - The details of the transformations will be exlained below.
+ - The details of the transformations will be explained below.
- Without adding transformation rules, you can still view the "[GitHub Metrics](/livedemo/DataSources/GitHub)" dashboard. However, if you want to view "[Weekly Bug Retro](/livedemo/QAEngineers/WeeklyBugRetro)", "[Weekly Community Retro](/livedemo/OSSMaintainers/WeeklyCommunityRetro)" or other pre-built dashboards, the following transformation rules, especially "Type/Bug", should be added.
- Each GitHub repo has at most ONE set of transformations.
@@ -152,7 +152,7 @@ You can also select "Not using Jobs in GitHub Action as Deployments" if you're n
#### Additional Settings (Optional)
-- Tags Limit: It'll compare the last N pairs of tags to get the "commit diff', "issue diff" between tags. N defaults to 10.
+- Tags Limit: It'll compare the last N pairs of tags to get the "commit diff", "issue diff" between tags. N defaults to 10.
- commit diff: new commits for a tag relative to the previous one
- issue diff: issues solved by the new commits for a tag relative to the previous one
@@ -166,7 +166,7 @@ Please click `Save` to save the transformation rules for the repo. In the data s
## Step 2 - Collect Data in a Project
### Step 2.1 - Create a Project
-Collecing GitHub data reuiqres creating a project first. You can visit the Project page from the side menu and create a new project by following the instructions on the user interface.
+Collecting GitHub data requires creating a project first. You can visit the Project page from the side menu and create a new project by following the instructions on the user interface.
@@ -179,7 +179,7 @@ Please note: if you don't see the repositories you are looking for, please check
### Step 2.3 - Set the Sync Policy
There are three settings for Sync Policy:
- Data Time Range: You can select the time range of the data you wish to collect. The default is set to the past six months.
-- Sync Frequency: You can choose how often you would like to sync your data in this step by selecting a sync frequency option or enter a cron code to specify your prefered schedule.
+- Sync Frequency: You can choose how often you would like to sync your data in this step by selecting a sync frequency option or enter a cron code to specify your preferred schedule.
- Skip Failed Tasks: sometime a few tasks may fail in a long pipeline; you can choose to skip them to avoid spending more time in running the pipeline all over again.
diff --git a/docs/Configuration/images/github-set-transformation2.png b/docs/Configuration/images/github-set-transformation2.png
index 672d9872b2..fc51dae334 100644
Binary files a/docs/Configuration/images/github-set-transformation2.png and b/docs/Configuration/images/github-set-transformation2.png differ
diff --git a/docs/Configuration/images/gitlab-set-transformation2.png b/docs/Configuration/images/gitlab-set-transformation2.png
index 3d8071b8c2..833b3c2f64 100644
Binary files a/docs/Configuration/images/gitlab-set-transformation2.png and b/docs/Configuration/images/gitlab-set-transformation2.png differ
diff --git a/docs/DataModels/DevLakeDomainLayerSchema.md b/docs/DataModels/DevLakeDomainLayerSchema.md
index 069d28b2b0..666c419711 100644
--- a/docs/DataModels/DevLakeDomainLayerSchema.md
+++ b/docs/DataModels/DevLakeDomainLayerSchema.md
@@ -41,7 +41,7 @@ This is the up-to-date domain layer schema for DevLake. Tables (entities) are ca
-When reading the schema, you'll notice that many tables' primary key is called `id`. Unlike auto-increment id or UUID, `id` is a string composed of several parts to uniquely identify similar entities (e.g. repo) from different platforms (e.g. Github/GitLab) and allow them to co-exist in a single table.
+When reading the schema, you'll notice that many tables' primary key is called `id`. Unlike auto-increment id or UUID, `id` is a string composed of several parts to uniquely identify similar entities (e.g. repo) from different platforms (e.g. GitHub/GitLab) and allow them to co-exist in a single table.
Tables that end with WIP are still under development.
@@ -66,38 +66,38 @@ Apache DevLake provides 2 plugins:
#### issues
-An `issue` is the abstraction of Github/GitLab/BitBucket/Jira/TAPD/Zentao... issues.
-
-| **field** | **type** | **length** | **description** [...]
-| :-------------------------- | :------- | :--------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| `id` | varchar | 255 | An issue's `id` is composed of < plugin >:< Entity >:< PK0 >[:PK1]..." <ul><li>For Github issues, a Github issue's id is like "github:GithubIssues:< GithubIssueId >". E.g. 'github:GithubIssues:1049355647'</li> <li>For Jira issues, a Github repo's id is like "jira:JiraIssues:< JiraSourceId >:< JiraIssueId >". E.g. 'jira:JiraIssues:1:10063'. < JiraSourceId > is used to identify which jira source the issue came from, since DevLake user [...]
-| `issue_key` | varchar | 255 | The key of this issue. For example, the key of this Github [issue](https://github.com/apache/incubator-devlake/issues/1145) is 1145. [...]
-| `url` | varchar | 255 | The url of the issue. It's a web address in most cases. [...]
-| `title` | varchar | 255 | The title of an issue [...]
-| `description` | longtext | | The detailed description/summary of an issue [...]
-| `type` | varchar | 100 | The standard type of this issue. There're 3 standard types: <ul><li>REQUIREMENT: this issue is a feature</li><li>BUG: this issue is a bug found during test</li><li>INCIDENT: this issue is a bug found after release</li></ul>The 3 standard types are transformed from the original types of an issue. The transformation rule is set in the '.env' file or 'config-ui' before data collection. For issues with an original type that has not mapp [...]
-| `original_type` | varchar | 100 | The original type of an issue. [...]
-| `status` | varchar | 100 | The standard statuses of this issue. There're 3 standard statuses: <ul><li> TODO: this issue is in backlog or to-do list</li><li>IN_PROGRESS: this issue is in progress</li><li>DONE: this issue is resolved or closed</li></ul>The 3 standard statuses are transformed from the original statuses of an issue. The transformation rule: <ul><li>For Jira issue status: transformed from the Jira issue's `statusCategory`. Jira issue has 3 default [...]
-| `original_status` | varchar | 100 | The original status of an issue. [...]
-| `story_point` | double | | The story point of this issue. Only certain types(e.g. story) of Jira or TAPD issues has story points [...]
-| `priority` | varchar | 255 | The priority of the issue [...]
-| `component` | varchar | 255 | The component a bug-issue affects. This field only supports Github plugin for now. The value is transformed from Github issue labels by the rules set according to the user's configuration of .env by end users during DevLake installation. [...]
-| `severity` | varchar | 255 | The severity level of a bug-issue. This field only supports Github plugin for now. The value is transformed from Github issue labels by the rules set according to the user's configuration of .env by end users during DevLake installation. [...]
-| `parent_issue_id` | varchar | 255 | The id of its parent issue [...]
-| `epic_key` | varchar | 255 | The key of the epic this issue belongs to. For tools with no epic-type issues such as Github and GitLab, this field is default to an empty string [...]
-| `original_estimate_minutes` | int | | The original estimation of the time allocated for this issue [...]
-| `time_spent_minutes` | int | | The original estimation of the time allocated for this issue [...]
-| `time_remaining_minutes` | int | | The remaining time to resolve the issue [...]
-| `creator_id` | varchar | 255 | The id of issue creator [...]
-| `creator_name` | varchar | 255 | The name of the creator [...]
-| `assignee_id` | varchar | 255 | The id of issue assignee.<ul><li>For Github issues: this is the last assignee of an issue if the issue has multiple assignees</li><li>For Jira issues: this is the assignee of the issue at the time of collection</li></ul> [...]
-| `assignee_name` | varchar | 255 | The name of the assignee [...]
-| `created_date` | datetime | 3 | The time issue created [...]
-| `updated_date` | datetime | 3 | The last time issue gets updated [...]
-| `resolution_date` | datetime | 3 | The time the issue changes to 'DONE'. [...]
-| `lead_time_minutes` | int | | Describes the cycle time from issue creation to issue resolution.<ul><li>For issues whose type = 'REQUIREMENT' and status = 'DONE', lead_time_minutes = resolution_date - created_date. The unit is minute.</li><li>For issues whose type != 'REQUIREMENT' or status != 'DONE', lead_time_minutes is null</li></ul> [...]
-| `original_project` | varchar | 255 | The name of the original project this issue belongs to. Transformed from a Jira project's name, a TAPD workspace's name, etc. [...]
-| `icon_url` | varchar | 255 | The url of the issue icon. [...]
+An `issue` is the abstraction of GitHub/GitLab/BitBucket/Jira/TAPD/Zentao... issues.
+
+| **field** | **type** | **length** | **description** [...]
+| :-------------------------- | :------- | :--------- |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| `id` | varchar | 255 | An issue's `id` is composed of < plugin >:< Entity >:< PK0 >[:PK1]..." <ul><li>For Github issues, a Github issue's id is like "github:GithubIssues:< GithubIssueId >". E.g. 'github:GithubIssues:1049355647'</li> <li>For Jira issues, a Github repo's id is like "jira:JiraIssues:< JiraSourceId >:< JiraIssueId >". E.g. 'jira:JiraIssues:1:10063'. < JiraSourceId > is used to identify which jira source the issue came from, since DevLake user [...]
+| `issue_key` | varchar | 255 | The key of this issue. For example, the key of this Github [issue](https://github.com/apache/incubator-devlake/issues/1145) is 1145. [...]
+| `url` | varchar | 255 | The url of the issue. It's a web address in most cases. [...]
+| `title` | varchar | 255 | The title of an issue [...]
+| `description` | longtext | | The detailed description/summary of an issue [...]
+| `type` | varchar | 100 | The standard type of this issue. There are 3 standard types: <ul><li>REQUIREMENT: this issue is a feature</li><li>BUG: this issue is a bug found during test</li><li>INCIDENT: this issue is a bug found after release</li></ul>The 3 standard types are transformed from the original types of an issue. The transformation rule is set in the '.env' file or 'config-ui' before data collection. For issues with an original type that has not map [...]
+| `original_type` | varchar | 100 | The original type of an issue. [...]
+| `status` | varchar | 100 | The standard statuses of this issue. There are 3 standard statuses: <ul><li> TODO: this issue is in backlog or to-do list</li><li>IN_PROGRESS: this issue is in progress</li><li>DONE: this issue is resolved or closed</li></ul>The 3 standard statuses are transformed from the original statuses of an issue. The transformation rule: <ul><li>For Jira issue status: transformed from the Jira issue's `statusCategory`. Jira issue has 3 defaul [...]
+| `original_status` | varchar | 100 | The original status of an issue. [...]
+| `story_point` | double | | The story point of this issue. Only certain types(e.g. story) of Jira or TAPD issues has story points [...]
+| `priority` | varchar | 255 | The priority of the issue [...]
+| `component` | varchar | 255 | The component a bug-issue affects. This field only supports Github plugin for now. The value is transformed from Github issue labels by the rules set according to the user's configuration of .env by end users during DevLake installation. [...]
+| `severity` | varchar | 255 | The severity level of a bug-issue. This field only supports Github plugin for now. The value is transformed from Github issue labels by the rules set according to the user's configuration of .env by end users during DevLake installation. [...]
+| `parent_issue_id` | varchar | 255 | The id of its parent issue [...]
+| `epic_key` | varchar | 255 | The key of the epic this issue belongs to. For tools with no epic-type issues such as Github and GitLab, this field is default to an empty string [...]
+| `original_estimate_minutes` | int | | The original estimation of the time allocated for this issue [...]
+| `time_spent_minutes` | int | | The original estimation of the time allocated for this issue [...]
+| `time_remaining_minutes` | int | | The remaining time to resolve the issue [...]
+| `creator_id` | varchar | 255 | The id of issue creator [...]
+| `creator_name` | varchar | 255 | The name of the creator [...]
+| `assignee_id` | varchar | 255 | The id of issue assignee.<ul><li>For Github issues: this is the last assignee of an issue if the issue has multiple assignees</li><li>For Jira issues: this is the assignee of the issue at the time of collection</li></ul> [...]
+| `assignee_name` | varchar | 255 | The name of the assignee [...]
+| `created_date` | datetime | 3 | The time issue created [...]
+| `updated_date` | datetime | 3 | The last time issue gets updated [...]
+| `resolution_date` | datetime | 3 | The time the issue changes to 'DONE'. [...]
+| `lead_time_minutes` | int | | Describes the cycle time from issue creation to issue resolution.<ul><li>For issues whose type = 'REQUIREMENT' and status = 'DONE', lead_time_minutes = resolution_date - created_date. The unit is minute.</li><li>For issues whose type != 'REQUIREMENT' or status != 'DONE', lead_time_minutes is null</li></ul> [...]
+| `original_project` | varchar | 255 | The name of the original project this issue belongs to. Transformed from a Jira project's name, a TAPD workspace's name, etc. [...]
+| `icon_url` | varchar | 255 | The url of the issue icon. [...]
#### issue_assignees
@@ -129,7 +129,7 @@ This table shows the comments of issues. Only GitHub and TAPD issue comments are
| `account_id` | varchar | 255 | The id of the account who made the comment | FK_accounts.id |
| `body` | longtext | | The body/detail of the comment | |
| `created_date` | datetime | 3 | The creation date of the comment | |
-| `updated_date` | datetime | 3 | The updation date of the comment | |
+| `updated_date` | datetime | 3 | The update date of the comment | |
#### issue_changelogs
@@ -263,7 +263,7 @@ GitHub, GitLab or BitBucket repositories.
#### repo_commits
-The commits belong to the history of a repository. More than one repos can share the same commits if one is a fork of the other.
+The commits belong to the history of a repository. More than one repo can share the same commits if one is a fork of the other.
| **field** | **type** | **length** | **description** | **key** |
| :----------- | :------- | :--------- | :-------------- | :------------- |
@@ -486,7 +486,7 @@ A cicd_pipeline is the abstraction of a top-level CI/CD execution, e.g. a GitHub
#### cicd_tasks
-A cicd_task is the abstraction of the bottom-level CI/CD excecution.
+A cicd_task is the abstraction of the bottom-level CI/CD execution.
- For GitHub: a cicd_task is a GitHub job run in a GitHub workflow run.
- For GitLab: a cicd_task is a GitLab job run of a GitLab pipeline run.
@@ -508,6 +508,35 @@ A cicd_task is the abstraction of the bottom-level CI/CD excecution.
| `finished_date` | datetime | 3 | When this task finished | | |
| `cicd_scope_id` | longtext | | The id of cicd_scope this task belongs to | FK_cicd_scopes.id |
+#### cicd_deployments
+A cicd_deployment refers to a deployment at the project level. In the case where a pipeline run or build deploys across three distinct repositories, it will be categorized as ONE cicd_deployment while being recorded as THREE separate cicd_deployment_commits.
+It may come from several sources:
+
+- Domain layer [cicd_pipelines](#cicd_pipelines), such as GitHub workflow runs, GitLab pipelines, Jenkins builds and BitBucket pipelines, etc. Deployments from cicd_pipelines will be transformed according to the regex configuration set in the Blueprint transformation before adding to this table.
+- Tool layer deployments: in v0.20, only the BitBucket\Bamboo\GitLab and GitHub(Use GraphQL APIs) plugins collect the independent deployment entity which you can find in table.\_tool_bitbucket_deployments and \_tool_bamboo_deploy_builds, there will be more in the future.
+- Deployments pushed directly from webhooks
+
+
+Additional Notes
+
+- CICD deployments are recorded at the project level, not the repository level.
+- CICD deployment commits are individual commits that represent a single deployment across multiple repositories.
+
+
+| **field** | **type** | **length** | **description** | **key** |
+|:----------------|:---------|:-----------|:---------------------------------------------------------------------------------------------------|:------------------|
+| `id` | varchar | 255 | The deployment_id of this deployment. The value will be set with `id` when it comes from webhooks. | PK |
+| `cicd_scope_id` | varchar | 255 | The id of cicd_scope this deployment belongs to | FK_cicd_scopes.id |
+| `name` | varchar | 255 | The name of the deployment | |
+| `result` | varchar | 100 | The result of the deployment, e.g. SUCCESS, FAILURE | |
+| `status` | varchar | 100 | The status of this deployment, e.g. IN_PROGRESS, DONE | |
+| `environment` | varchar | 255 | The environment to deploy, only 'PRODUCTION' deployment will appear in v0.17 | |
+| `created_date` | datetime | 3 | The created time of the deployment. Deprecated. | |
+| `started_date` | datetime | 3 | The started time of the deployment. | |
+| `finished_date` | datetime | 3 | The finished time of the deployment | |
+| `duration_sec` | bigint | | The time this deployment takes | |
+
+
#### cicd_deployment_commits
A cicd_deployment_commit is a deployment in a specific repo. A deployment may come from several sources:
@@ -521,23 +550,23 @@ You can query deployments from this table by `SELECT DISTINCT cicd_deployment_id
Normally, one deployment only deploy to one repo. But in some cases, one deployment may deploy in multiple repos with different commits. In these cases, there will be multiple pairs of deployment-commit-repo, appeared in multiple entries in this table.
| **field** | **type** | **length** | **description** | **key** |
-| :---------------------------------- | :------- | :--------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------- |
+|:------------------------------------|:---------|:-----------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------|
| `id` | varchar | 255 | This key is the combination of the deployment's id and repo_url, e.g. <br/>- from a GitHub workflow run: github:GithubRun:1:384111310:3521097091:https://github.com/apache/incubator-devlake <br/>- from a Jenkins build, jenkins:JenkinsBuild:1:deploy#7:https://github.com/apache/incubator-devlake <br/>- from a webhook, webhook:1:90489d3951711d72:e6bde456807818c5c78d7b265964d6d48b653af6 | PK |
| `cicd_scope_id` | varchar | 255 | The id of cicd_scope this deployment_commit belongs to | FK_cicd_scopes.id |
| `cicd_deployment_id` | varchar | 255 | The deployment_id of this deployment_commit. The value will be set with `id` when it comes from webhooks. | |
| `name` | varchar | 255 | The name of the deployment | |
| `result` | varchar | 100 | The result of the deployment, e.g. SUCCESS, FAILURE | |
| `status` | varchar | 100 | The status of this deployment, e.g. IN_PROGRESS, DONE | |
-| `environment` | varchar | 255 | The environment to deploy, only 'PRODUCTION' deployment will appear in v0.17 | |
-| `created_date` | datetime | 3 | The created time of the deployment. Deprecated. | |
-| `started_date` | datetime | 3 | The started time of the deployment. | |
+| `environment` | varchar | 255 | The environment to deploy | |
+| `created_date` | datetime | 3 | The created time of the deployment. Deprecated. | |
+| `started_date` | datetime | 3 | The started time of the deployment. | |
| `finished_date` | datetime | 3 | The finished time of the deployment | |
| `duration_sec` | bigint | | The time this deployment takes | |
| `commit_sha` | char | 40 | The commit sha that triggers the deployment | |
| `ref_name` | varchar | 255 | The ref (branch/tag) name of the commit | |
| `repo_id` | varchar | 255 | - | |
| `repo_url` | varchar | 191 | The url of the repo | |
-| `prev_success_deployment_commit_id` | varchar | 255 | The last successful deployment_commit_id before this one in the same `cicd_scope`, `repo` and `environment`, which is used to calculate the new commits deployed by this deployment, thereby measuring [DORA - change lead time](../Metrics/LeadTimeForChanges.md). [...]
+| `prev_success_deployment_commit_id` | varchar | 255 | The last successful deployment_commit_id before this one in the same `cicd_scope`, `repo` and `environment`, which is used to calculate the new commits deployed by this deployment, thereby measuring [DORA - change lead time](../Metrics/LeadTimeForChanges.md). | |
### Domain 5 - Code Quality
@@ -631,7 +660,7 @@ The names of tables in the 'Code Quality' domain will start with a prefix cq\_
These entities are used to map entities between different domains. They are the key players to break data isolation.
-There're low-level entities such as issue_commits, users, and higher-level cross domain entities such as board_repos
+There are low-level entities such as issue_commits, users, and higher-level cross domain entities such as board_repos
#### issue_commits
diff --git a/docs/Overview/SupportedDataSources.md b/docs/Overview/SupportedDataSources.md
index 4b95aaf95d..2c1861a808 100644
--- a/docs/Overview/SupportedDataSources.md
+++ b/docs/Overview/SupportedDataSources.md
@@ -33,59 +33,61 @@ This table shows the entities collected by each plugin. Domain layer entities in
✅ : Collect by default.
💪 : Collect not by default. You need to add the corresponding subtasks to collect these entities in the [advanced mode](../Configuration/AdvancedMode.md).
-| Domain Layer Entities | ae | dora | gitextractor | incoming webhook | github | gitlab | jenkins | jira | refdiff | tapd | sonarqube | bamboo | azuredevops |
-| ------------------------------------------------------------------------------------------- | --- | ---- | ------------ | ---------------- | ------ | ------ | ------- | ---- | ------- | ---- | --------- | ------ | ----------- |
-| [accounts](../DataModels/DevLakeDomainLayerSchema.md/#accounts) | | | | | ✅ | ✅ | | ✅ | | ✅ | ✅ | | |
-| [board_issues](../DataModels/DevLakeDomainLayerSchema.md/#board_issues) | | | | | ✅ | ✅ | | ✅ | | ✅ | | | |
-| [board_repos](../DataModels/DevLakeDomainLayerSchema.md/#board_repos) | | | | | ✅ | ✅ | | | | | | | |
-| [board_sprints](../DataModels/DevLakeDomainLayerSchema.md/#board_sprints) | | | | | ✅ | | | ✅ | | ✅ | | | |
-| [boards](../DataModels/DevLakeDomainLayerSchema.md/#boards) | | | | | ✅ | ✅ | | ✅ | | ✅ | | | |
-| [cicd_pipeline_commits](../DataModels/DevLakeDomainLayerSchema.md/#cicd_pipeline_commits) | | ✅ | | | ✅ | ✅ | ✅ | | | | | ✅ | ✅ |
-| [cicd_pipelines](../DataModels/DevLakeDomainLayerSchema.md/#cicd_pipelines) | | ✅ | | | ✅ | ✅ | ✅ | | | | | ✅ | ✅ |
-| [cicd_scopes](../DataModels/DevLakeDomainLayerSchema.md/#cicd_scopes) | | ✅ | | | ✅ | ✅ | ✅ | | | | | ✅ | ✅ |
-| [cicd_tasks](../DataModels/DevLakeDomainLayerSchema.md/#cicd_tasks) | | ✅ | | 💪 | ✅ | ✅ | ✅ | | | | | ✅ | ✅ |
-| [commit_file_components](../DataModels/DevLakeDomainLayerSchema.md/#commit_file_components) | | | ✅ | | | | | | | | | | |
-| [commit_files](../DataModels/DevLakeDomainLayerSchema.md/#commit_files) | | | ✅ | | | | | | | | | | |
-| [commit_line_change](../DataModels/DevLakeDomainLayerSchema.md/#commit_line_change) | | | ✅ | | | | | | | | | | |
-| [commit_parents](../DataModels/DevLakeDomainLayerSchema.md/#commit_parents) | | | ✅ | | | | | | | | | | |
-| [commits](../DataModels/DevLakeDomainLayerSchema.md/#commits) | ✅ | | ✅ | | 💪 | 💪 | | | | | | | |
-| [commits_diffs](../DataModels/DevLakeDomainLayerSchema.md/#commits_diffs) | | | | | | | | | ✅ | | | | |
-| [components](../DataModels/DevLakeDomainLayerSchema.md/#components) | | | | | | | | | | | | | |
-| [finished_commits_diffs](../DataModels/DevLakeDomainLayerSchema.md/#finished_commits_diffs) | | | | | | | | | | | | | |
-| [issue_changelogs](../DataModels/DevLakeDomainLayerSchema.md/#issue_changelogs) | | | | | | | | ✅ | | ✅ | | | |
-| [issue_comments](../DataModels/DevLakeDomainLayerSchema.md/#issue_comments) | | | | | ✅ | | | | | ✅ | | | |
-| [issue_commits](../DataModels/DevLakeDomainLayerSchema.md/#issue_commits) | | | | | | | | ✅ | | ✅ | | | |
-| [issue_labels](../DataModels/DevLakeDomainLayerSchema.md/#issue_labels) | | | | | ✅ | ✅ | | | | ✅ | | | |
-| [issue_repo_commits](../DataModels/DevLakeDomainLayerSchema.md/#issue_repo_commits) | | | | | | | | ✅ | | | | | |
-| [issue_worklogs](../DataModels/DevLakeDomainLayerSchema.md/#issue_worklogs) | | | | | | | | ✅ | | ✅ | | | |
-| [issues](../DataModels/DevLakeDomainLayerSchema.md/#issues) | | | | | ✅ | | | ✅ | | ✅ | | | |
-| [project_issue_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_issue_metrics) | | ✅ | | | ✅ | ✅ | | ✅ | | ✅ | | | |
-| [project_mapping](../DataModels/DevLakeDomainLayerSchema.md/#project_mapping) | | ✅ | | | ✅ | ✅ | ✅ | ✅ | | ✅ | | | |
-| [project_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_metrics) | | ✅ | | | ✅ | ✅ | ✅ | ✅ | | ✅ | | | |
-| [project_pr_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_pr_metrics) | | ✅ | | | ✅ | ✅ | | | | ✅ | | | |
-| [project](../DataModels/DevLakeDomainLayerSchema.md/#project) | | ✅ | | | ✅ | ✅ | ✅ | ✅ | | ✅ | | | |
-| [pull_request_comments](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_comments) | | | | | ✅ | ✅ | | | | | | | |
-| [pull_request_commits](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_commits) | | | | | ✅ | ✅ | | | | | | | ✅ |
-| [pull_request_issues](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_issues) | | | | | ✅ | | | | | | | | |
-| [pull_request_labels](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_labels) | | | | | ✅ | ✅ | | | | | | | |
-| [pull_requests](../DataModels/DevLakeDomainLayerSchema.md/#pull_requests) | | | | | ✅ | ✅ | | | | | | | ✅ |
-| [ref_commits](../DataModels/DevLakeDomainLayerSchema.md/#ref_commits) | | | | | | | | | ✅ | | | | |
-| [refs](../DataModels/DevLakeDomainLayerSchema.md/#refs) | | | ✅ | | | | | | ✅ | | | | |
-| [refs_issues_diffs](../DataModels/DevLakeDomainLayerSchema.md/#refs_issues_diffs) | | | | | | | | | ✅ | | | | |
-| [ref_pr_cherry_picks](../DataModels/DevLakeDomainLayerSchema.md/#ref_pr_cherry_picks) | | | | | | | | | ✅ | | | | |
-| [repo_commits](../DataModels/DevLakeDomainLayerSchema.md/#repo_commits) | | | ✅ | | 💪 | 💪 | | | | | | | |
-| [repo_snapshot](../DataModels/DevLakeDomainLayerSchema.md/#repo_snapshot) | | | ✅ | | | | | | | | | | |
-| [repos](../DataModels/DevLakeDomainLayerSchema.md/#repos) | | | | | ✅ | ✅ | | | | | | | |
-| [sprint_issues](../DataModels/DevLakeDomainLayerSchema.md/#sprint_issues) | | | | | ✅ | | | ✅ | | ✅ | | | |
-| [sprints](../DataModels/DevLakeDomainLayerSchema.md/#sprints) | | | | | ✅ | | | ✅ | | ✅ | | | |
-| [team_users](../DataModels/DevLakeDomainLayerSchema.md/#team_users) | | | | | | | | | | | | | |
-| [teams](../DataModels/DevLakeDomainLayerSchema.md/#teams) | | | | | | | | | | | | | |
-| [user_account](../DataModels/DevLakeDomainLayerSchema.md/#user_accounts) | | | | | | | | | | | | | |
-| [users](../DataModels/DevLakeDomainLayerSchema.md/#users) | | | | | | | | ✅ | | ✅ | | | |
-| [cq_projects](../DataModels/DevLakeDomainLayerSchema.md/#cq_projects) | | | | | | | | | | | ✅ | | |
-| [cq_issues](../DataModels/DevLakeDomainLayerSchema.md/#cq_issues) | | | | | | | | | | | ✅ | | |
-| [cq_issue_code_blocks](../DataModels/DevLakeDomainLayerSchema.md/#cq_issue_code_blocks) | | | | | | | | | | | ✅ | | |
-| [cq_file_metrics](../DataModels/DevLakeDomainLayerSchema.md/#cq_file_metrics) | | | | | | | | | | | ✅ | | |
+| Domain Layer Entities | ae | dora | gitextractor | incoming webhook | GitHub | GitLab | Jenkins | Jira | refdiff | TAPD | SonarQube | Bamboo | Azure DevOps | BitBucket |
+|-----------------------------------------------------------------------------------------------|----|------|--------------|------------------|--------|--------|---------|------|---------|------|-----------|--------|--------------|-----------|
+| [accounts](../DataModels/DevLakeDomainLayerSchema.md/#accounts) | | | | | ✅ | ✅ | | ✅ | | ✅ | ✅ | | | |
+| [board_issues](../DataModels/DevLakeDomainLayerSchema.md/#board_issues) | | | | | ✅ | ✅ | | ✅ | | ✅ | | | | |
+| [board_repos](../DataModels/DevLakeDomainLayerSchema.md/#board_repos) | | | | | ✅ | ✅ | | | | | | | | |
+| [board_sprints](../DataModels/DevLakeDomainLayerSchema.md/#board_sprints) | | | | | ✅ | | | ✅ | | ✅ | | | | |
+| [boards](../DataModels/DevLakeDomainLayerSchema.md/#boards) | | | | | ✅ | ✅ | | ✅ | | ✅ | | | | |
+| [cicd_deployment_commits](../DataModels/DevLakeDomainLayerSchema.md/#cicd_deployment_commits) | | | | | ✅ | ✅ | | | | | | ✅ | | ✅ |
+| [cicd_deployments](../DataModels/DevLakeDomainLayerSchema.md/#cicd_deployments) | | ✅ | | | ✅ | ✅ | | | | | | ✅ | | ✅ |
+| [cicd_pipeline_commits](../DataModels/DevLakeDomainLayerSchema.md/#cicd_pipeline_commits) | | ✅ | | | ✅ | ✅ | ✅ | | | | | ✅ | ✅ | |
+| [cicd_pipelines](../DataModels/DevLakeDomainLayerSchema.md/#cicd_pipelines) | | ✅ | | | ✅ | ✅ | ✅ | | | | | ✅ | ✅ | |
+| [cicd_scopes](../DataModels/DevLakeDomainLayerSchema.md/#cicd_scopes) | | ✅ | | | ✅ | ✅ | ✅ | | | | | ✅ | ✅ | |
+| [cicd_tasks](../DataModels/DevLakeDomainLayerSchema.md/#cicd_tasks) | | ✅ | | 💪 | ✅ | ✅ | ✅ | | | | | ✅ | ✅ | |
+| [commit_file_components](../DataModels/DevLakeDomainLayerSchema.md/#commit_file_components) | | | ✅ | | | | | | | | | | | |
+| [commit_files](../DataModels/DevLakeDomainLayerSchema.md/#commit_files) | | | ✅ | | | | | | | | | | | |
+| [commit_line_change](../DataModels/DevLakeDomainLayerSchema.md/#commit_line_change) | | | ✅ | | | | | | | | | | | |
+| [commit_parents](../DataModels/DevLakeDomainLayerSchema.md/#commit_parents) | | | ✅ | | | | | | | | | | | |
+| [commits](../DataModels/DevLakeDomainLayerSchema.md/#commits) | ✅ | | ✅ | | 💪 | 💪 | | | | | | | | |
+| [commits_diffs](../DataModels/DevLakeDomainLayerSchema.md/#commits_diffs) | | | | | | | | | ✅ | | | | | |
+| [components](../DataModels/DevLakeDomainLayerSchema.md/#components) | | | | | | | | | | | | | | |
+| [finished_commits_diffs](../DataModels/DevLakeDomainLayerSchema.md/#finished_commits_diffs) | | | | | | | | | | | | | | |
+| [issue_changelogs](../DataModels/DevLakeDomainLayerSchema.md/#issue_changelogs) | | | | | | | | ✅ | | ✅ | | | | |
+| [issue_comments](../DataModels/DevLakeDomainLayerSchema.md/#issue_comments) | | | | | ✅ | | | | | ✅ | | | | |
+| [issue_commits](../DataModels/DevLakeDomainLayerSchema.md/#issue_commits) | | | | | | | | ✅ | | ✅ | | | | |
+| [issue_labels](../DataModels/DevLakeDomainLayerSchema.md/#issue_labels) | | | | | ✅ | ✅ | | | | ✅ | | | | |
+| [issue_repo_commits](../DataModels/DevLakeDomainLayerSchema.md/#issue_repo_commits) | | | | | | | | ✅ | | | | | | |
+| [issue_worklogs](../DataModels/DevLakeDomainLayerSchema.md/#issue_worklogs) | | | | | | | | ✅ | | ✅ | | | | |
+| [issues](../DataModels/DevLakeDomainLayerSchema.md/#issues) | | | | | ✅ | | | ✅ | | ✅ | | | | |
+| [project_issue_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_issue_metrics) | | ✅ | | | ✅ | ✅ | | ✅ | | ✅ | | | | |
+| [project_mapping](../DataModels/DevLakeDomainLayerSchema.md/#project_mapping) | | ✅ | | | ✅ | ✅ | ✅ | ✅ | | ✅ | | | | |
+| [project_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_metrics) | | ✅ | | | ✅ | ✅ | ✅ | ✅ | | ✅ | | | | |
+| [project_pr_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_pr_metrics) | | ✅ | | | ✅ | ✅ | | | | ✅ | | | | |
+| [project](../DataModels/DevLakeDomainLayerSchema.md/#project) | | ✅ | | | ✅ | ✅ | ✅ | ✅ | | ✅ | | | | |
+| [pull_request_comments](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_comments) | | | | | ✅ | ✅ | | | | | | | | |
+| [pull_request_commits](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_commits) | | | | | ✅ | ✅ | | | | | | | ✅ | |
+| [pull_request_issues](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_issues) | | | | | ✅ | | | | | | | | | |
+| [pull_request_labels](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_labels) | | | | | ✅ | ✅ | | | | | | | | |
+| [pull_requests](../DataModels/DevLakeDomainLayerSchema.md/#pull_requests) | | | | | ✅ | ✅ | | | | | | | ✅ | |
+| [ref_commits](../DataModels/DevLakeDomainLayerSchema.md/#ref_commits) | | | | | | | | | ✅ | | | | | |
+| [refs](../DataModels/DevLakeDomainLayerSchema.md/#refs) | | | ✅ | | | | | | ✅ | | | | | |
+| [refs_issues_diffs](../DataModels/DevLakeDomainLayerSchema.md/#refs_issues_diffs) | | | | | | | | | ✅ | | | | | |
+| [ref_pr_cherry_picks](../DataModels/DevLakeDomainLayerSchema.md/#ref_pr_cherry_picks) | | | | | | | | | ✅ | | | | | |
+| [repo_commits](../DataModels/DevLakeDomainLayerSchema.md/#repo_commits) | | | ✅ | | 💪 | 💪 | | | | | | | | |
+| [repo_snapshot](../DataModels/DevLakeDomainLayerSchema.md/#repo_snapshot) | | | ✅ | | | | | | | | | | | |
+| [repos](../DataModels/DevLakeDomainLayerSchema.md/#repos) | | | | | ✅ | ✅ | | | | | | | | |
+| [sprint_issues](../DataModels/DevLakeDomainLayerSchema.md/#sprint_issues) | | | | | ✅ | | | ✅ | | ✅ | | | | |
+| [sprints](../DataModels/DevLakeDomainLayerSchema.md/#sprints) | | | | | ✅ | | | ✅ | | ✅ | | | | |
+| [team_users](../DataModels/DevLakeDomainLayerSchema.md/#team_users) | | | | | | | | | | | | | | |
+| [teams](../DataModels/DevLakeDomainLayerSchema.md/#teams) | | | | | | | | | | | | | | |
+| [user_account](../DataModels/DevLakeDomainLayerSchema.md/#user_accounts) | | | | | | | | | | | | | | |
+| [users](../DataModels/DevLakeDomainLayerSchema.md/#users) | | | | | | | | ✅ | | ✅ | | | | |
+| [cq_projects](../DataModels/DevLakeDomainLayerSchema.md/#cq_projects) | | | | | | | | | | | ✅ | | | |
+| [cq_issues](../DataModels/DevLakeDomainLayerSchema.md/#cq_issues) | | | | | | | | | | | ✅ | | | |
+| [cq_issue_code_blocks](../DataModels/DevLakeDomainLayerSchema.md/#cq_issue_code_blocks) | | | | | | | | | | | ✅ | | | |
+| [cq_file_metrics](../DataModels/DevLakeDomainLayerSchema.md/#cq_file_metrics) | | | | | | | | | | | ✅ | | | |
## Data Sync Policy
diff --git a/static/img/ConfigUI/gitlab-set-data-scope.png b/static/img/ConfigUI/gitlab-set-data-scope.png
index 2e976b2083..c493d13596 100644
Binary files a/static/img/ConfigUI/gitlab-set-data-scope.png and b/static/img/ConfigUI/gitlab-set-data-scope.png differ