You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by he...@apache.org on 2023/02/08 03:41:12 UTC
[incubator-devlake-website] branch main updated: docs(cicd): update docs (#421)
This is an automated email from the ASF dual-hosted git repository.
hez pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
The following commit(s) were added to refs/heads/main by this push:
new 66ec39bf8f docs(cicd): update docs (#421)
66ec39bf8f is described below
commit 66ec39bf8f6031284d59318ed32b2be069aabeb3
Author: Warren Chen <yi...@merico.dev>
AuthorDate: Wed Feb 8 11:41:07 2023 +0800
docs(cicd): update docs (#421)
* docs(cicd): update docs
* fix for review
---
docs/DataModels/DevLakeDomainLayerSchema.md | 61 +++++++++++++--------
docs/UserManuals/ConfigUI/GitHub.md | 7 ++-
docs/UserManuals/ConfigUI/Jenkins.md | 14 +++--
static/img/ConfigUI/github-action-job.png | Bin 0 -> 132479 bytes
static/img/ConfigUI/github-action-run.png | Bin 0 -> 112303 bytes
static/img/ConfigUI/jenkins-job-build-stage.png | Bin 0 -> 244394 bytes
.../DataModels/DevLakeDomainLayerSchema.md | 57 ++++++++++++-------
.../version-v0.15/UserManuals/ConfigUI/GitHub.md | 6 +-
.../version-v0.15/UserManuals/ConfigUI/Jenkins.md | 15 +++--
9 files changed, 107 insertions(+), 53 deletions(-)
diff --git a/docs/DataModels/DevLakeDomainLayerSchema.md b/docs/DataModels/DevLakeDomainLayerSchema.md
index 519d54e94e..78c0539ec8 100644
--- a/docs/DataModels/DevLakeDomainLayerSchema.md
+++ b/docs/DataModels/DevLakeDomainLayerSchema.md
@@ -151,14 +151,14 @@ This table shows the work logged under issues. Usually, an issue has multiple wo
A `board` is an issue list or a collection of issues. It's the abstraction of a Jira board, a Jira project, a [GitHub issue list](https://github.com/apache/incubator-devlake/issues) or a GitLab issue list. This table can be used to filter issues by the boards they belong to.
-| **field** | **type** | **length** | **description** | **key** |
-| :------------- | :------- | :--------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------ |
-| `id` | varchar | 255 | A board's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..." <ul><li>For a Github repo's issue list, the board id is like "< github >:< GithubRepos >:< GithubRepoId >". Eg. "github:GithubRepo:384111310"</li> <li>For a Jira Board, the id is like the board id is like "< jira >:< JiraSourceId >< JiraBoards >:< JiraBoardsId >". Eg. "jira:1:JiraBoards:12"</li></ul> | PK |
-| `name` | varchar | 255 | The name of the board. Note: the board name of a Github project 'apache/incubator-devlake' is 'apache/incubator-devlake', representing the [default issue list](https://github.com/apache/incubator-devlake/issues). | |
-| `description` | varchar | 255 | The description of the board. | |
-| `url` | varchar | 255 | The url of the board. Eg. https://github.com/apache/incubator-devlake | |
-| `created_date` | datetime | 3 | Board creation time | |
-| `type` | varchar | 255 | Identify scrum and non-scrum board | |
+| **field** | **type** | **length** | **description** | **key** |
+| :------------- | :------- | :--------- |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :------ |
+| `id` | varchar | 255 | A board's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..." <ul><li>For a Github repo's issue list, the board id is like "< github >:< GithubRepos >:< ConnectionId >:< GithubRepoId >". Eg. "github:GithubRepo:384111310"</li> <li>For a Jira Board, the id is like the board id is like "< jira >:< JiraSourceId >< JiraBoards >:< ConnectionId >:< JiraBoardsId >". Eg. "jira:1:JiraBoards:1:12"</li></ul> | PK |
+| `name` | varchar | 255 | The name of the board. Note: the board name of a Github project 'apache/incubator-devlake' is 'apache/incubator-devlake', representing the [default issue list](https://github.com/apache/incubator-devlake/issues). | |
+| `description` | varchar | 255 | The description of the board. | |
+| `url` | varchar | 255 | The url of the board. Eg. https://github.com/apache/incubator-devlake | |
+| `created_date` | datetime | 3 | Board creation time | |
+| `type` | varchar | 255 | Identify scrum and non-scrum board | |
#### board_issues
@@ -213,18 +213,18 @@ This table shows the relation between sprints and issues that have been added to
Information about GitHub or Gitlab repositories. A repository is always owned by a user.
-| **field** | **type** | **length** | **description** | **key** |
-| :------------- | :------- | :--------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------- |
-| `id` | varchar | 255 | A repo's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<br/>For example, a Github repo's id is like "< github >:< GithubRepos >< GithubRepoId >". Eg. 'github:GithubRepos:384111310' | PK |
-| `name` | varchar | 255 | The name of repo. | |
-| `description` | varchar | 255 | The description of repo. | |
-| `url` | varchar | 255 | The url of repo. Eg. https://github.com/apache/incubator-devlake | |
-| `owner_id` | varchar | 255 | The id of the owner of repo | FK_accounts.id |
-| `language` | varchar | 255 | The major language of repo. Eg. The language for apache/incubator-devlake is 'Go' | |
-| `forked_from` | varchar | 255 | Empty unless the repo is a fork in which case it contains the `id` of the repo the repo is forked from. | |
-| `deleted` | tinyint | 255 | 0: repo is active 1: repo has been deleted | |
-| `created_date` | datetime | 3 | Repo creation date | |
-| `updated_date` | datetime | 3 | Last full update was done for this repo | |
+| **field** | **type** | **length** | **description** | **key** |
+| :------------- | :------- | :--------- |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :------------- |
+| `id` | varchar | 255 | A repo's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<br/>For example, a Github repo's id is like "< github >:< GithubRepos >:< ConnectionId >:< GithubRepoId >". Eg. 'github:GithubRepos:1:384111310' | PK |
+| `name` | varchar | 255 | The name of repo. | |
+| `description` | varchar | 255 | The description of repo. | |
+| `url` | varchar | 255 | The url of repo. Eg. https://github.com/apache/incubator-devlake | |
+| `owner_id` | varchar | 255 | The id of the owner of repo | FK_accounts.id |
+| `language` | varchar | 255 | The major language of repo. Eg. The language for apache/incubator-devlake is 'Go' | |
+| `forked_from` | varchar | 255 | Empty unless the repo is a fork in which case it contains the `id` of the repo the repo is forked from. | |
+| `deleted` | tinyint | 255 | 0: repo is active 1: repo has been deleted | |
+| `created_date` | datetime | 3 | Repo creation date | |
+| `updated_date` | datetime | 3 | Last full update was done for this repo | |
#### repo_languages(WIP)
@@ -435,6 +435,23 @@ Events of pull requests.
### Domain 4 - CI/CD(WIP)
+#### cicd_scopes
+
+Information about Jenkins Job, GitHub Action or Gitlab CI.
+- For GitHub: a GitHub repo is converted to a cicd_scope
+- For Jenkins: a GitLab project is converted to a cicd_scope
+- For GitLab: a Jenkins job is converted to a cicd_scope
+
+| **field** | **type** | **length** | **description** | **key** |
+| :------------- | :------- | :--------- |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :------------- |
+| `id` | varchar | 255 | A cicd_scope's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<br/>For example, a Github cicd_scope's id is like "< github >:< GithubRepos >:< ConnectionId >:< GithubRepoId >". Eg. 'github:GithubRepos:1:384111310' | PK |
+| `name` | varchar | 255 | The name of cicd_scope. | |
+| `description` | varchar | 255 | The description of cicd_scope. | |
+| `url` | varchar | 255 | The url of cicd_scope. Eg. https://github.com/apache/incubator-devlake or https://jenkins.xxx.cn/view/PROD/job/OPS_releasev2/ | |
+| `created_date` | datetime | 3 | cicd_scope creation date | |
+| `updated_date` | datetime | 3 | Date of the last data collection for this cicd_scope | |
+
+
#### cicd_pipelines
A cicd_pipeline is a series of builds that have connections or a standalone build.
@@ -481,7 +498,7 @@ A cicd_task is a single job of ci/cd.
### Project Metric Entities
-#### project_pr_metrics
+#### project_pr_metrics
| **field** | **type** | **length** | **description** | **key** |
| :-------- | :-------- |:-----------|:---------------------------------------------------------------------------------------| :-------- |
@@ -649,7 +666,7 @@ import "github.com/apache/incubator-devlake/models/domainlayer/domaininfo"
domaininfo := domaininfo.GetDomainTablesInfo()
for _, table := range domaininfo {
- // do something
+// do something
}
```
diff --git a/docs/UserManuals/ConfigUI/GitHub.md b/docs/UserManuals/ConfigUI/GitHub.md
index 4cb53afc1d..040ed52d49 100644
--- a/docs/UserManuals/ConfigUI/GitHub.md
+++ b/docs/UserManuals/ConfigUI/GitHub.md
@@ -120,7 +120,12 @@ If you're using GitHub Action to conduct `deployments`, please select "Detect De
- Deployment: A GitHub Action job with a name that matches the given regEx will be considered as a deployment.
- Production: A GitHub Action job with a name that matches the given regEx will be considered a job in the production environment.
-By the above two fields, DevLake can identify a production deployment among massive CI jobs.
+A GitHub workflow run has many jobs. Each GitHub workflow run is converted to a
+cicd_pipeline in the domain layer and each GitHub Action job is converted to a cicd_task in the domain layer.
+
+
+
+The deployment and production regex is always applied to the records in the cicd_tasks table.
You can also select "Not using Jobs in GitHub Action as Deployments" if you're not using GitHub action to conduct deployments.
diff --git a/docs/UserManuals/ConfigUI/Jenkins.md b/docs/UserManuals/ConfigUI/Jenkins.md
index 41ba2d6e51..999a9e04c9 100644
--- a/docs/UserManuals/ConfigUI/Jenkins.md
+++ b/docs/UserManuals/ConfigUI/Jenkins.md
@@ -54,12 +54,18 @@ Jenkins only supports `CI/CD` domain entities, transformed from Jenkins builds a
This set of configurations is used for calculating [DORA metrics](../DORA.md).
-If you're using Jenkins builds to conduct `deployments`, please select "Detect Deployment from Jenkins Builds", and input the RegEx in the following fields:
+If you'd like to define `deployments` with Jenkins, please select "Detect Deployment from Jenkins Builds", and provide the following regexes
-- Deployment: A Jenkins build with a name that matches the given regEx will be considered as a deployment.
-- Production: A Jenkins build with a name that matches the given regEx will be considered a build in the production environment.
+- Deployment: Jenkins stages whose names match the regex will be registered as deployments (if a Jenkins build has no stage, its job name will be used to match the regex)
+- Production: Jenkins stages whose names match the regex will be assigned environment 'PRODUCTION' (if a Jenkins build has no stage, its job name will be used to match the regex)
-By the above two fields, DevLake can identify a production deployment among massive CI jobs.
+This is how it works behind the scene:
+
+- If a Jenkins build has stages, it's converted to a cicd_pipeline and its stages are converted to cicd_tasks in DevLake's domain layer schema.
+- If a Jenkins build has no stage, it's converted to both a cicd_pipeline and a cicd_task whose names are the build's jobName.
+
+After the conversion, the two regexes are applied to the records in the cicd_tasks table.
+
You can also select "Not using Jenkins builds as Deployments" if you're not using Jenkins to conduct deployments.
diff --git a/static/img/ConfigUI/github-action-job.png b/static/img/ConfigUI/github-action-job.png
new file mode 100644
index 0000000000..de6847e39f
Binary files /dev/null and b/static/img/ConfigUI/github-action-job.png differ
diff --git a/static/img/ConfigUI/github-action-run.png b/static/img/ConfigUI/github-action-run.png
new file mode 100644
index 0000000000..a7ee280c1d
Binary files /dev/null and b/static/img/ConfigUI/github-action-run.png differ
diff --git a/static/img/ConfigUI/jenkins-job-build-stage.png b/static/img/ConfigUI/jenkins-job-build-stage.png
new file mode 100644
index 0000000000..0f5879b277
Binary files /dev/null and b/static/img/ConfigUI/jenkins-job-build-stage.png differ
diff --git a/versioned_docs/version-v0.15/DataModels/DevLakeDomainLayerSchema.md b/versioned_docs/version-v0.15/DataModels/DevLakeDomainLayerSchema.md
index 519d54e94e..5450c74868 100644
--- a/versioned_docs/version-v0.15/DataModels/DevLakeDomainLayerSchema.md
+++ b/versioned_docs/version-v0.15/DataModels/DevLakeDomainLayerSchema.md
@@ -151,14 +151,14 @@ This table shows the work logged under issues. Usually, an issue has multiple wo
A `board` is an issue list or a collection of issues. It's the abstraction of a Jira board, a Jira project, a [GitHub issue list](https://github.com/apache/incubator-devlake/issues) or a GitLab issue list. This table can be used to filter issues by the boards they belong to.
-| **field** | **type** | **length** | **description** | **key** |
-| :------------- | :------- | :--------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------ |
-| `id` | varchar | 255 | A board's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..." <ul><li>For a Github repo's issue list, the board id is like "< github >:< GithubRepos >:< GithubRepoId >". Eg. "github:GithubRepo:384111310"</li> <li>For a Jira Board, the id is like the board id is like "< jira >:< JiraSourceId >< JiraBoards >:< JiraBoardsId >". Eg. "jira:1:JiraBoards:12"</li></ul> | PK |
-| `name` | varchar | 255 | The name of the board. Note: the board name of a Github project 'apache/incubator-devlake' is 'apache/incubator-devlake', representing the [default issue list](https://github.com/apache/incubator-devlake/issues). | |
-| `description` | varchar | 255 | The description of the board. | |
-| `url` | varchar | 255 | The url of the board. Eg. https://github.com/apache/incubator-devlake | |
-| `created_date` | datetime | 3 | Board creation time | |
-| `type` | varchar | 255 | Identify scrum and non-scrum board | |
+| **field** | **type** | **length** | **description** | **key** |
+| :------------- | :------- | :--------- |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :------ |
+| `id` | varchar | 255 | A board's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..." <ul><li>For a Github repo's issue list, the board id is like "< github >:< GithubRepos >:< ConnectionId >:< GithubRepoId >". Eg. "github:GithubRepo:384111310"</li> <li>For a Jira Board, the id is like the board id is like "< jira >:< JiraSourceId >< JiraBoards >:< ConnectionId >:< JiraBoardsId >". Eg. "jira:1:JiraBoards:1:12"</li></ul> | PK |
+| `name` | varchar | 255 | The name of the board. Note: the board name of a Github project 'apache/incubator-devlake' is 'apache/incubator-devlake', representing the [default issue list](https://github.com/apache/incubator-devlake/issues). | |
+| `description` | varchar | 255 | The description of the board. | |
+| `url` | varchar | 255 | The url of the board. Eg. https://github.com/apache/incubator-devlake | |
+| `created_date` | datetime | 3 | Board creation time | |
+| `type` | varchar | 255 | Identify scrum and non-scrum board | |
#### board_issues
@@ -213,18 +213,18 @@ This table shows the relation between sprints and issues that have been added to
Information about GitHub or Gitlab repositories. A repository is always owned by a user.
-| **field** | **type** | **length** | **description** | **key** |
-| :------------- | :------- | :--------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------- |
-| `id` | varchar | 255 | A repo's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<br/>For example, a Github repo's id is like "< github >:< GithubRepos >< GithubRepoId >". Eg. 'github:GithubRepos:384111310' | PK |
-| `name` | varchar | 255 | The name of repo. | |
-| `description` | varchar | 255 | The description of repo. | |
-| `url` | varchar | 255 | The url of repo. Eg. https://github.com/apache/incubator-devlake | |
-| `owner_id` | varchar | 255 | The id of the owner of repo | FK_accounts.id |
-| `language` | varchar | 255 | The major language of repo. Eg. The language for apache/incubator-devlake is 'Go' | |
-| `forked_from` | varchar | 255 | Empty unless the repo is a fork in which case it contains the `id` of the repo the repo is forked from. | |
-| `deleted` | tinyint | 255 | 0: repo is active 1: repo has been deleted | |
-| `created_date` | datetime | 3 | Repo creation date | |
-| `updated_date` | datetime | 3 | Last full update was done for this repo | |
+| **field** | **type** | **length** | **description** | **key** |
+| :------------- | :------- | :--------- |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :------------- |
+| `id` | varchar | 255 | A repo's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<br/>For example, a Github repo's id is like "< github >:< GithubRepos >:< ConnectionId >:< GithubRepoId >". Eg. 'github:GithubRepos:1:384111310' | PK |
+| `name` | varchar | 255 | The name of repo. | |
+| `description` | varchar | 255 | The description of repo. | |
+| `url` | varchar | 255 | The url of repo. Eg. https://github.com/apache/incubator-devlake | |
+| `owner_id` | varchar | 255 | The id of the owner of repo | FK_accounts.id |
+| `language` | varchar | 255 | The major language of repo. Eg. The language for apache/incubator-devlake is 'Go' | |
+| `forked_from` | varchar | 255 | Empty unless the repo is a fork in which case it contains the `id` of the repo the repo is forked from. | |
+| `deleted` | tinyint | 255 | 0: repo is active 1: repo has been deleted | |
+| `created_date` | datetime | 3 | Repo creation date | |
+| `updated_date` | datetime | 3 | Last full update was done for this repo | |
#### repo_languages(WIP)
@@ -435,6 +435,23 @@ Events of pull requests.
### Domain 4 - CI/CD(WIP)
+#### cicd_scopes
+
+Information about Jenkins Job, GitHub Action or Gitlab CI.
+- For GitHub: a GitHub repo is converted to a cicd_scope
+- For Jenkins: a GitLab project is converted to a cicd_scope
+- For GitLab: a Jenkins job is converted to a cicd_scope
+
+| **field** | **type** | **length** | **description** | **key** |
+| :------------- | :------- | :--------- |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :------------- |
+| `id` | varchar | 255 | A cicd_scope's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<br/>For example, a Github cicd_scope's id is like "< github >:< GithubRepos >:< ConnectionId >:< GithubRepoId >". Eg. 'github:GithubRepos:1:384111310' | PK |
+| `name` | varchar | 255 | The name of cicd_scope. | |
+| `description` | varchar | 255 | The description of cicd_scope. | |
+| `url` | varchar | 255 | The url of cicd_scope. Eg. https://github.com/apache/incubator-devlake or https://jenkins.xxx.cn/view/PROD/job/OPS_releasev2/ | |
+| `created_date` | datetime | 3 | cicd_scope creation date | |
+| `updated_date` | datetime | 3 | Date of the last data collection for this cicd_scope | |
+
+
#### cicd_pipelines
A cicd_pipeline is a series of builds that have connections or a standalone build.
diff --git a/versioned_docs/version-v0.15/UserManuals/ConfigUI/GitHub.md b/versioned_docs/version-v0.15/UserManuals/ConfigUI/GitHub.md
index 4cb53afc1d..c80152260f 100644
--- a/versioned_docs/version-v0.15/UserManuals/ConfigUI/GitHub.md
+++ b/versioned_docs/version-v0.15/UserManuals/ConfigUI/GitHub.md
@@ -120,7 +120,11 @@ If you're using GitHub Action to conduct `deployments`, please select "Detect De
- Deployment: A GitHub Action job with a name that matches the given regEx will be considered as a deployment.
- Production: A GitHub Action job with a name that matches the given regEx will be considered a job in the production environment.
-By the above two fields, DevLake can identify a production deployment among massive CI jobs.
+A GitHub workflow run has many jobs. Each GitHub workflow run is converted to a cicd_pipeline in the domain layer and each GitHub Action job is converted to a cicd_task in the domain layer.
+
+
+
+The deployment and production regex is always applied to the records in the cicd_tasks table.
You can also select "Not using Jobs in GitHub Action as Deployments" if you're not using GitHub action to conduct deployments.
diff --git a/versioned_docs/version-v0.15/UserManuals/ConfigUI/Jenkins.md b/versioned_docs/version-v0.15/UserManuals/ConfigUI/Jenkins.md
index 41ba2d6e51..260348e53e 100644
--- a/versioned_docs/version-v0.15/UserManuals/ConfigUI/Jenkins.md
+++ b/versioned_docs/version-v0.15/UserManuals/ConfigUI/Jenkins.md
@@ -51,15 +51,20 @@ Jenkins only supports `CI/CD` domain entities, transformed from Jenkins builds a
- CI/CD: Jenkins builds, stages, etc.
### Step 3 - Adding Transformation Rules (Optional)
-
This set of configurations is used for calculating [DORA metrics](../DORA.md).
-If you're using Jenkins builds to conduct `deployments`, please select "Detect Deployment from Jenkins Builds", and input the RegEx in the following fields:
+If you'd like to define `deployments` with Jenkins, please select "Detect Deployment from Jenkins Builds", and provide the following regexes
+
+- Deployment: Jenkins stages whose names match the regex will be registered as deployments (if a Jenkins build has no stage, its job name will be used to match the regex)
+- Production: Jenkins stages whose names match the regex will be assigned environment 'PRODUCTION' (if a Jenkins build has no stage, its job name will be used to match the regex)
+
+This is how it works behind the scene:
-- Deployment: A Jenkins build with a name that matches the given regEx will be considered as a deployment.
-- Production: A Jenkins build with a name that matches the given regEx will be considered a build in the production environment.
+- If a Jenkins build has stages, it's converted to a cicd_pipeline and its stages are converted to cicd_tasks in DevLake's domain layer schema.
+- If a Jenkins build has no stage, it's converted to both a cicd_pipeline and a cicd_task whose names are the build's jobName.
-By the above two fields, DevLake can identify a production deployment among massive CI jobs.
+After the conversion, the two regexes are applied to the records in the cicd_tasks table.
+
You can also select "Not using Jenkins builds as Deployments" if you're not using Jenkins to conduct deployments.