You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by zk...@apache.org on 2022/12/01 03:40:50 UTC
[incubator-devlake-website] branch main updated (0af470f85 -> ea568f1e1)
This is an automated email from the ASF dual-hosted git repository.
zky pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
from 0af470f85 fix: update the pic name for pr merge rate
new d1075c24b docs: update refdiff docs
new c1a03418b docs: update refdiff docs
new ea568f1e1 docs: update refdiff docs
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/DataModels/DevLakeDomainLayerSchema.md | 319 ++---
docs/Plugins/refdiff.md | 86 +-
.../domain_layer_model_v0.14.ndm2 | 1457 +++++---------------
static/img/DomainLayerSchema/schema-diagram.png | Bin 542870 -> 503803 bytes
4 files changed, 544 insertions(+), 1318 deletions(-)
[incubator-devlake-website] 03/03: docs: update refdiff docs
Posted by zk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
commit ea568f1e19d3b2848c457ba4658e8919ca2b186f
Author: abeizn <zi...@merico.dev>
AuthorDate: Mon Nov 28 16:39:14 2022 +0800
docs: update refdiff docs
---
docs/Plugins/refdiff.md | 6 +-
.../domain_layer_model_v0.14.ndm2 | 166 ++++++++++-----------
static/img/DomainLayerSchema/schema-diagram.png | Bin 503846 -> 503803 bytes
3 files changed, 86 insertions(+), 86 deletions(-)
diff --git a/docs/Plugins/refdiff.md b/docs/Plugins/refdiff.md
index 53ec882c2..5e86ff6c7 100644
--- a/docs/Plugins/refdiff.md
+++ b/docs/Plugins/refdiff.md
@@ -8,8 +8,8 @@ description: >
RefDiff is a plugin that performs calculation tasks and has 2 main purposes.
-- Calculate the difference in commits between releases/tags to [analyze the amount of code in each release](add an anchor link)
-- Calculate the difference in commits between deployments to [calculate DORA metrics](add an anchor link)
+- Calculate the difference in commits between releases/tags to [analyze the amount of code in each release](https://github.com/apache/incubator-devlake/blob/main/plugins/refdiff/tasks/commit_diff_calculator.go)
+- Calculate the difference in commits between deployments to [calculate DORA metrics](https://github.com/apache/incubator-devlake/blob/main/plugins/refdiff/tasks/project_deployment_commit_diff_calculator.go)
And the output of RefDiff is stored in the table commits_diffs, finished_commits_diffs, ref_commits.
@@ -39,7 +39,7 @@ To trigger the enrichment, you need to insert a new task into your pipeline.
github:GithubRepo:1:384111310:refs/tags/v0.6.1 TAG
```
2. If you want to run calculatePrCherryPick, please configure GITHUB_PR_TITLE_PATTERN in .env, you can check the example in .env.example(we have a default value, please make sure your pattern is disclosed by single quotes '')
-3. And then, trigger a pipeline like the following format, you can also define sub-tasks, calculateRefDiff will calculate commits between two refs, and creatRefBugStats will create a table to show a bug list between two refs:
+3. And then, trigger a pipeline like the following format:
```shell
curl 'http://localhost:8080/pipelines' \
diff --git a/static/img/DomainLayerSchema/domain_layer_model_v0.14.ndm2 b/static/img/DomainLayerSchema/domain_layer_model_v0.14.ndm2
index 197f3d1b9..72654c5f7 100644
--- a/static/img/DomainLayerSchema/domain_layer_model_v0.14.ndm2
+++ b/static/img/DomainLayerSchema/domain_layer_model_v0.14.ndm2
@@ -10019,7 +10019,7 @@
},
{
"objectType": "Table_MYSQL",
- "name": "ref_commits",
+ "name": "finished_commits_diffs",
"comment": "",
"engine": "",
"characterSet": "",
@@ -10118,58 +10118,6 @@
"virtualExpr": "",
"virtualType": "",
"oldName": "old_commit_sha"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "new_ref_id",
- "type": "char",
- "length": 255,
- "decimals": -2147483648,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": false,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "new_ref_id"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "olde_red_id",
- "type": "char",
- "length": 255,
- "decimals": -2147483648,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": false,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "olde_red_id"
}
],
"indexes": [],
@@ -10179,14 +10127,14 @@
"fields": [
{
"objectType": "IndexField_MYSQL",
- "name": "new_ref_id",
+ "name": "new_commit_sha",
"keyLength": 0,
"order": "",
"oldName": ""
},
{
"objectType": "IndexField_MYSQL",
- "name": "olde_red_id",
+ "name": "old_commit_sha",
"keyLength": 0,
"order": "",
"oldName": ""
@@ -10199,12 +10147,12 @@
"foreignKeys": [
{
"objectType": "ForeignKey_MYSQL",
- "name": "fk_ref_commits_ref_commits_1",
+ "name": "fk_commits_diffs_status_commits_diffs_status_1",
"fields": [
"new_commit_sha"
],
"referenceSchema": "Default",
- "referenceTable": "finished_commits_diffs",
+ "referenceTable": "commits_diffs",
"referenceFields": [
"new_commit_sha"
],
@@ -10216,12 +10164,12 @@
},
{
"objectType": "ForeignKey_MYSQL",
- "name": "fk_ref_commits_ref_commits_2",
+ "name": "fk_commits_diffs_status_commits_diffs_status_2",
"fields": [
"old_commit_sha"
],
"referenceSchema": "Default",
- "referenceTable": "finished_commits_diffs",
+ "referenceTable": "commits_diffs",
"referenceFields": [
"old_commit_sha"
],
@@ -10230,6 +10178,23 @@
"sourceCardinality": "NoneRelationship",
"targetCardinality": "NoneRelationship",
"oldName": ""
+ },
+ {
+ "objectType": "ForeignKey_MYSQL",
+ "name": "fk_commits_diffs_status_commits_diffs_status_3",
+ "fields": [
+ "old_commit_sha"
+ ],
+ "referenceSchema": "lake",
+ "referenceTable": "cicd_pipeline_commits",
+ "referenceFields": [
+ "commit_sha"
+ ],
+ "onDelete": "",
+ "onUpdate": "",
+ "sourceCardinality": "NoneRelationship",
+ "targetCardinality": "NoneRelationship",
+ "oldName": ""
}
],
"checks": [],
@@ -10238,7 +10203,7 @@
},
{
"objectType": "Table_MYSQL",
- "name": "finished_commits_diffs",
+ "name": "ref_commits",
"comment": "",
"engine": "",
"characterSet": "",
@@ -10337,6 +10302,58 @@
"virtualExpr": "",
"virtualType": "",
"oldName": "old_commit_sha"
+ },
+ {
+ "objectType": "TableField_MYSQL",
+ "name": "new_ref_id",
+ "type": "char",
+ "length": 255,
+ "decimals": -2147483648,
+ "isUnsigned": false,
+ "isZeroFill": false,
+ "setEnumValues": "",
+ "isBinary": false,
+ "charset": "",
+ "collation": "",
+ "isNullable": false,
+ "defaultType": "Others",
+ "defaultValue": "",
+ "isOnUpdateCurrentTimestamp": false,
+ "isAutoInc": false,
+ "comment": "",
+ "columnFormat": "",
+ "storage": "",
+ "isVirtual": false,
+ "isGeneratedAlways": false,
+ "virtualExpr": "",
+ "virtualType": "",
+ "oldName": "new_ref_id"
+ },
+ {
+ "objectType": "TableField_MYSQL",
+ "name": "old_ref_id",
+ "type": "char",
+ "length": 255,
+ "decimals": -2147483648,
+ "isUnsigned": false,
+ "isZeroFill": false,
+ "setEnumValues": "",
+ "isBinary": false,
+ "charset": "",
+ "collation": "",
+ "isNullable": false,
+ "defaultType": "Others",
+ "defaultValue": "",
+ "isOnUpdateCurrentTimestamp": false,
+ "isAutoInc": false,
+ "comment": "",
+ "columnFormat": "",
+ "storage": "",
+ "isVirtual": false,
+ "isGeneratedAlways": false,
+ "virtualExpr": "",
+ "virtualType": "",
+ "oldName": "old_ref_id"
}
],
"indexes": [],
@@ -10346,14 +10363,14 @@
"fields": [
{
"objectType": "IndexField_MYSQL",
- "name": "new_commit_sha",
+ "name": "new_ref_id",
"keyLength": 0,
"order": "",
"oldName": ""
},
{
"objectType": "IndexField_MYSQL",
- "name": "old_commit_sha",
+ "name": "old_ref_id",
"keyLength": 0,
"order": "",
"oldName": ""
@@ -10366,12 +10383,12 @@
"foreignKeys": [
{
"objectType": "ForeignKey_MYSQL",
- "name": "fk_commits_diffs_status_commits_diffs_status_1",
+ "name": "fk_ref_commits_ref_commits_1",
"fields": [
"new_commit_sha"
],
"referenceSchema": "Default",
- "referenceTable": "commits_diffs",
+ "referenceTable": "finished_commits_diffs",
"referenceFields": [
"new_commit_sha"
],
@@ -10383,12 +10400,12 @@
},
{
"objectType": "ForeignKey_MYSQL",
- "name": "fk_commits_diffs_status_commits_diffs_status_2",
+ "name": "fk_ref_commits_ref_commits_2",
"fields": [
"old_commit_sha"
],
"referenceSchema": "Default",
- "referenceTable": "commits_diffs",
+ "referenceTable": "finished_commits_diffs",
"referenceFields": [
"old_commit_sha"
],
@@ -10397,23 +10414,6 @@
"sourceCardinality": "NoneRelationship",
"targetCardinality": "NoneRelationship",
"oldName": ""
- },
- {
- "objectType": "ForeignKey_MYSQL",
- "name": "fk_commits_diffs_status_commits_diffs_status_3",
- "fields": [
- "old_commit_sha"
- ],
- "referenceSchema": "lake",
- "referenceTable": "cicd_pipeline_commits",
- "referenceFields": [
- "commit_sha"
- ],
- "onDelete": "",
- "onUpdate": "",
- "sourceCardinality": "NoneRelationship",
- "targetCardinality": "NoneRelationship",
- "oldName": ""
}
],
"checks": [],
diff --git a/static/img/DomainLayerSchema/schema-diagram.png b/static/img/DomainLayerSchema/schema-diagram.png
index af04b66e5..fa50e1d4b 100644
Binary files a/static/img/DomainLayerSchema/schema-diagram.png and b/static/img/DomainLayerSchema/schema-diagram.png differ
[incubator-devlake-website] 02/03: docs: update refdiff docs
Posted by zk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
commit c1a03418bb627192579ba0f14f7d74a592351fc1
Author: abeizn <zi...@merico.dev>
AuthorDate: Thu Nov 24 20:39:53 2022 +0800
docs: update refdiff docs
---
docs/DataModels/DevLakeDomainLayerSchema.md | 319 ++---
docs/Plugins/refdiff.md | 7 +-
.../domain_layer_model_v0.14.ndm2 | 1459 +++++---------------
static/img/DomainLayerSchema/schema-diagram.png | Bin 542870 -> 503846 bytes
4 files changed, 501 insertions(+), 1284 deletions(-)
diff --git a/docs/DataModels/DevLakeDomainLayerSchema.md b/docs/DataModels/DevLakeDomainLayerSchema.md
index 3164480ee..d9074022e 100644
--- a/docs/DataModels/DevLakeDomainLayerSchema.md
+++ b/docs/DataModels/DevLakeDomainLayerSchema.md
@@ -13,31 +13,30 @@ Referring to DevLake's [architecture](../Overview/Architecture.md), the data in
Domain layer schema itself includes 2 logical layers: a `DWD` layer and a `DWM` layer. The DWD layer stores the detailed data points, while the DWM is the slight aggregation and operation of DWD to store more organized details or middle-level metrics.
-
## Use Cases
+
1. [All metrics](../Metrics) from pre-built dashboards are based on this data schema.
2. As a user, you can create your own customized dashboards based on this data schema.
3. As a contributor, you can refer to this data schema while working on the ETL logic when adding/updating data source plugins.
-
## Data Models
This is the up-to-date domain layer schema for DevLake v0.10.x. Tables (entities) are categorized into 5 domains.
+
1. Issue tracking domain entities: Jira issues, GitHub issues, GitLab issues, etc.
2. Source code management domain entities: Git/GitHub/Gitlab commits and refs(tags and branches), etc.
3. Code review domain entities: GitHub PRs, Gitlab MRs, etc.
4. CI/CD domain entities: Jenkins jobs & builds, etc.
5. Cross-domain entities: entities that map entities from different domains to break data isolation.
-
### Schema Diagram
+
[](/img/DomainLayerSchema/schema-diagram.png)
When reading the schema, you'll notice that many tables' primary key is called `id`. Unlike auto-increment id or UUID, `id` is a string composed of several parts to uniquely identify similar entities (e.g. repo) from different platforms (e.g. Github/Gitlab) and allow them to co-exist in a single table.
Tables that end with WIP are still under development.
-
### Naming Conventions
1. The name of a table is in plural form. Eg. boards, issues, etc.
@@ -45,13 +44,14 @@ Tables that end with WIP are still under development.
3. Value of the field in enum type are in capital letters. Eg. [table.issues.type](#issues) has 3 values, REQUIREMENT, BUG, INCIDENT. Values that are phrases, such as 'IN_PROGRESS' of [table.issues.status](#issues), are separated with underscore '\_'.
## How to Customize Data Models
+
Apache DevLake provides 2 plugins:
+
- [customize](https://devlake.apache.org/docs/Plugins/customize): to create/delete columns in the domain layer schema with the data extracted from [raw layer tables](https://devlake.apache.org/docs/Overview/Architecture/#dataflow)
- [dbt](https://devlake.apache.org/docs/Plugins/customize): to transform data based on the domain layer schema and generate new tables
<br/>
-
## DWD Entities - (Data Warehouse Detail)
### Domain 1 - Issue Tracking
@@ -61,9 +61,9 @@ Apache DevLake provides 2 plugins:
An `issue` is the abstraction of Jira/Github/GitLab/TAPD/... issues.
| **field** | **type** | **length** | **description** [...]
-| :-------------------------- | :------- | :--------- |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| :-------------------------- | :------- | :--------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
| `id` | varchar | 255 | An issue's `id` is composed of < plugin >:< Entity >:< PK0 >[:PK1]..." <ul><li>For Github issues, a Github issue's id is like "github:GithubIssues:< GithubIssueId >". Eg. 'github:GithubIssues:1049355647'</li> <li>For Jira issues, a Github repo's id is like "jira:JiraIssues:< JiraSourceId >:< JiraIssueId >". Eg. 'jira:JiraIssues:1:10063'. < JiraSourceId > is used to identify which jira source the issue came from, since DevLake users [...]
-| `issue_key` | varchar | 255 | The key of this issue. For example, the key of this Github [issue](https://github.com/apache/incubator-devlake/issues/1145) is 1145. [...]
+| `issue_key` | varchar | 255 | The key of this issue. For example, the key of this Github [issue](https://github.com/apache/incubator-devlake/issues/1145) is 1145. [...]
| `url` | varchar | 255 | The url of the issue. It's a web address in most cases. [...]
| `title` | varchar | 255 | The title of an issue [...]
| `description` | longtext | | The detailed description/summary of an issue [...]
@@ -77,9 +77,9 @@ An `issue` is the abstraction of Jira/Github/GitLab/TAPD/... issues.
| `parent_issue_id` | varchar | 255 | The id of its parent issue [...]
| `epic_key` | varchar | 255 | The key of the epic this issue belongs to. For tools with no epic-type issues such as Github and Gitlab, this field is default to an empty string [...]
| `original_estimate_minutes` | int | | The original estimation of the time allocated for this issue [...]
-| `time_spent_minutes` | int | | The original estimation of the time allocated for this issue [...]
-| `time_remaining_minutes` | int | | The remaining time to resolve the issue [...]
-| `creator_id` | varchar | 255 | The id of issue creator [...]
+| `time_spent_minutes` | int | | The original estimation of the time allocated for this issue [...]
+| `time_remaining_minutes` | int | | The remaining time to resolve the issue [...]
+| `creator_id` | varchar | 255 | The id of issue creator [...]
| `creator_name` | varchar | 255 | The name of the creator [...]
| `assignee_id` | varchar | 255 | The id of issue assignee.<ul><li>For Github issues: this is the last assignee of an issue if the issue has multiple assignees</li><li>For Jira issues: this is the assignee of the issue at the time of collection</li></ul> [...]
| `assignee_name` | varchar | 255 | The name of the assignee [...]
@@ -87,7 +87,7 @@ An `issue` is the abstraction of Jira/Github/GitLab/TAPD/... issues.
| `updated_date` | datetime | 3 | The last time issue gets updated [...]
| `resolution_date` | datetime | 3 | The time the issue changes to 'DONE'. [...]
| `lead_time_minutes` | int | | Describes the cycle time from issue creation to issue resolution.<ul><li>For issues whose type = 'REQUIREMENT' and status = 'DONE', lead_time_minutes = resolution_date - created_date. The unit is minute.</li><li>For issues whose type != 'REQUIREMENT' or status != 'DONE', lead_time_minutes is null</li></ul> [...]
-| `deployment_id` | varchar | 255 | The cicd_task that relates to this issue [...]
+| `deployment_id` | varchar | 255 | The cicd_task that relates to this issue [...]
#### issue_labels
@@ -98,19 +98,18 @@ This table shows the labels of issues. Multiple entries can exist per issue. Thi
| `name` | varchar | 255 | Label name | |
| `issue_id` | varchar | 255 | Issue ID | FK_issues.id |
-
#### issue_comments(WIP)
This table shows the comments of issues. Issues with multiple comments are shown as multiple records. This table can be used to calculate _metric - issue response time_.
-| **field** | **type** | **length** | **description** | **key** |
-| :------------- | :------- | :--------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------- |
-| `id` | varchar | 255 | The unique id of a comment | PK |
-| `issue_id` | varchar | 255 | Issue ID | FK_issues.id |
-| `account_id` | varchar | 255 | The id of the account who made the comment | FK_accounts.id |
-| `body` | longtext | | The body/detail of the comment | |
-| `created_date` | datetime | 3 | The creation date of the comment | |
-| `updated_date` | datetime | 3 | The last time comment gets updated | |
+| **field** | **type** | **length** | **description** | **key** |
+| :------------- | :------- | :--------- | :----------------------------------------- | :------------- |
+| `id` | varchar | 255 | The unique id of a comment | PK |
+| `issue_id` | varchar | 255 | Issue ID | FK_issues.id |
+| `account_id` | varchar | 255 | The id of the account who made the comment | FK_accounts.id |
+| `body` | longtext | | The body/detail of the comment | |
+| `created_date` | datetime | 3 | The creation date of the comment | |
+| `updated_date` | datetime | 3 | The last time comment gets updated | |
#### issue_changelogs
@@ -130,21 +129,19 @@ This table shows the changelogs of issues. Issues with multiple changelogs are s
| `to_value` | varchar | 255 | The transformed/standardized new value of the changed field | |
| `created_date` | datetime | 3 | The creation date of the changelog | |
-
#### issue_worklogs
This table shows the work logged under issues. Usually, an issue has multiple worklogs logged by different developers.
-| **field** | **type** | **length** | **description** | **key** |
-| :------------------- | :------- | :--------- | :------------------------------------------------------------------------------------------- |:----------------|
-| `id` | varchar | 255 | The id of the worklog | PK |
-| `author_id` | varchar | 255 | The id of the author who logged the work | FK_accounts.id |
-| `comment` | longtext | 255 | The comment made while logging the work. | |
-| `time_spent_minutes` | int | | The time logged. The unit of value is normalized to minute. Eg. 1d =) 480, 4h30m =) 270 | |
-| `logged_date` | datetime | 3 | The time of this logging action | |
-| `started_date` | datetime | 3 | Start time of the worklog | |
-| `issue_id` | varchar | 255 | Issue ID | FK_issues.id |
-
+| **field** | **type** | **length** | **description** | **key** |
+| :------------------- | :------- | :--------- | :-------------------------------------------------------------------------------------- | :------------- |
+| `id` | varchar | 255 | The id of the worklog | PK |
+| `author_id` | varchar | 255 | The id of the author who logged the work | FK_accounts.id |
+| `comment` | longtext | 255 | The comment made while logging the work. | |
+| `time_spent_minutes` | int | | The time logged. The unit of value is normalized to minute. Eg. 1d =) 480, 4h30m =) 270 | |
+| `logged_date` | datetime | 3 | The time of this logging action | |
+| `started_date` | datetime | 3 | Start time of the worklog | |
+| `issue_id` | varchar | 255 | Issue ID | FK_issues.id |
#### boards
@@ -153,11 +150,11 @@ A `board` is an issue list or a collection of issues. It's the abstraction of a
| **field** | **type** | **length** | **description** | **key** |
| :------------- | :------- | :--------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------ |
| `id` | varchar | 255 | A board's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..." <ul><li>For a Github repo's issue list, the board id is like "< github >:< GithubRepos >:< GithubRepoId >". Eg. "github:GithubRepo:384111310"</li> <li>For a Jira Board, the id is like the board id is like "< jira >:< JiraSourceId >< JiraBoards >:< JiraBoardsId >". Eg. "jira:1:JiraBoards:12"</li></ul> | PK |
-| `name` | varchar | 255 | The name of the board. Note: the board name of a Github project 'apache/incubator-devlake' is 'apache/incubator-devlake', representing the [default issue list](https://github.com/apache/incubator-devlake/issues). | |
+| `name` | varchar | 255 | The name of the board. Note: the board name of a Github project 'apache/incubator-devlake' is 'apache/incubator-devlake', representing the [default issue list](https://github.com/apache/incubator-devlake/issues). | |
| `description` | varchar | 255 | The description of the board. | |
-| `url` | varchar | 255 | The url of the board. Eg. https://github.com/apache/incubator-devlake | |
-| `created_date` | datetime | 3 | Board creation time | |
-| `type` | varchar | 255 | Identify scrum and non-scrum board | |
+| `url` | varchar | 255 | The url of the board. Eg. https://github.com/apache/incubator-devlake | |
+| `created_date` | datetime | 3 | Board creation time | |
+| `type` | varchar | 255 | Identify scrum and non-scrum board | |
#### board_issues
@@ -172,16 +169,16 @@ This table shows the relation between boards and issues. This table can be used
A `sprint` is the abstraction of Jira sprints, TAPD iterations and GitHub milestones. A sprint contains a list of issues.
-| **field** | **type** | **length** | **description** [...]
-| :------------------ | :------- | :--------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
-| `id` | varchar | 255 | A sprint's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<ul><li>A sprint in a Github repo is a milestone, the sprint id is like "< github >:< GithubRepos >:< GithubRepoId >:< milestoneNumber >".<br/>Eg. The id for this [sprint](https://github.com/apache/incubator-devlake/milestone/5) is "github:GithubRepo:384111310:5"</li><li>For a Jira Board, the id is like "< jira >:< JiraSourceId >< JiraBoards >:< JiraBoardsId >".<br/>Eg. [...]
-| `name` | varchar | 255 | The name of sprint.<br/>For Github projects, the sprint name is the milestone name. For instance, 'v0.10.0 - Introduce Temporal to DevLake' is the name of this [sprint](https://github.com/apache/incubator-devlake/milestone/5). [...]
-| `url` | varchar | 255 | The url of sprint. [...]
-| `status` | varchar | 255 | There're 3 statuses of a sprint:<ul><li>CLOSED: a completed sprint</li><li>ACTIVE: a sprint started but not completed</li><li>FUTURE: a sprint that has not started</li></ul> [...]
-| `started_date` | datetime | 3 | The start time of a sprint [...]
-| `ended_date` | datetime | 3 | The planned/estimated end time of a sprint. It's usually set when planning a sprint. [...]
-| `completed_date` | datetime | 3 | The actual time to complete a sprint. [...]
-| `original_board_id` | datetime | 3 | The id of board where the sprint first created. This field is not null only when this entity is transformed from Jira sprints.<br/>In Jira, sprint and board entities have 2 types of relation:<ul><li>A sprint is created based on a specific board. In this case, board(1):(n)sprint. The `original_board_id` is used to show the relation.</li><li>A sprint can be mapped to multiple boards, a board can also show multiple sprints. In this case, board [...]
+| **field** | **type** | **length** | **description** [...]
+| :------------------ | :------- | :--------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| `id` | varchar | 255 | A sprint's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<ul><li>A sprint in a Github repo is a milestone, the sprint id is like "< github >:< GithubRepos >:< GithubRepoId >:< milestoneNumber >".<br/>Eg. The id for this [sprint](https://github.com/apache/incubator-devlake/milestone/5) is "github:GithubRepo:384111310:5"</li><li>For a Jira Board, the id is like "< jira >:< JiraSourceId >< JiraBoards >:< JiraBoardsId >".<br/>Eg. [...]
+| `name` | varchar | 255 | The name of sprint.<br/>For Github projects, the sprint name is the milestone name. For instance, 'v0.10.0 - Introduce Temporal to DevLake' is the name of this [sprint](https://github.com/apache/incubator-devlake/milestone/5). [...]
+| `url` | varchar | 255 | The url of sprint. [...]
+| `status` | varchar | 255 | There're 3 statuses of a sprint:<ul><li>CLOSED: a completed sprint</li><li>ACTIVE: a sprint started but not completed</li><li>FUTURE: a sprint that has not started</li></ul> [...]
+| `started_date` | datetime | 3 | The start time of a sprint [...]
+| `ended_date` | datetime | 3 | The planned/estimated end time of a sprint. It's usually set when planning a sprint. [...]
+| `completed_date` | datetime | 3 | The actual time to complete a sprint. [...]
+| `original_board_id` | datetime | 3 | The id of board where the sprint first created. This field is not null only when this entity is transformed from Jira sprints.<br/>In Jira, sprint and board entities have 2 types of relation:<ul><li>A sprint is created based on a specific board. In this case, board(1):(n)sprint. The `original_board_id` is used to show the relation.</li><li>A sprint can be mapped to multiple boards, a board can also show multiple sprints. In this case, board [...]
#### sprint_issues
@@ -212,18 +209,18 @@ This table shows the relation between sprints and issues that have been added to
Information about GitHub or Gitlab repositories. A repository is always owned by a user.
-| **field** | **type** | **length** | **description** | **key** |
-| :------------- | :------- | :--------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------- |
-| `id` | varchar | 255 | A repo's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<br/>For example, a Github repo's id is like "< github >:< GithubRepos >< GithubRepoId >". Eg. 'github:GithubRepos:384111310' | PK |
-| `name` | varchar | 255 | The name of repo. | |
-| `description` | varchar | 255 | The description of repo. | |
-| `url` | varchar | 255 | The url of repo. Eg. https://github.com/apache/incubator-devlake | |
-| `owner_id` | varchar | 255 | The id of the owner of repo | FK_accounts.id |
-| `language` | varchar | 255 | The major language of repo. Eg. The language for apache/incubator-devlake is 'Go' | |
-| `forked_from` | varchar | 255 | Empty unless the repo is a fork in which case it contains the `id` of the repo the repo is forked from. | |
-| `deleted` | tinyint | 255 | 0: repo is active 1: repo has been deleted | |
-| `created_date` | datetime | 3 | Repo creation date | |
-| `updated_date` | datetime | 3 | Last full update was done for this repo | |
+| **field** | **type** | **length** | **description** | **key** |
+| :------------- | :------- | :--------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------- |
+| `id` | varchar | 255 | A repo's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<br/>For example, a Github repo's id is like "< github >:< GithubRepos >< GithubRepoId >". Eg. 'github:GithubRepos:384111310' | PK |
+| `name` | varchar | 255 | The name of repo. | |
+| `description` | varchar | 255 | The description of repo. | |
+| `url` | varchar | 255 | The url of repo. Eg. https://github.com/apache/incubator-devlake | |
+| `owner_id` | varchar | 255 | The id of the owner of repo | FK_accounts.id |
+| `language` | varchar | 255 | The major language of repo. Eg. The language for apache/incubator-devlake is 'Go' | |
+| `forked_from` | varchar | 255 | Empty unless the repo is a fork in which case it contains the `id` of the repo the repo is forked from. | |
+| `deleted` | tinyint | 255 | 0: repo is active 1: repo has been deleted | |
+| `created_date` | datetime | 3 | Repo creation date | |
+| `updated_date` | datetime | 3 | Last full update was done for this repo | |
#### repo_languages(WIP)
@@ -234,7 +231,7 @@ The table is filled in when the repo has been first inserted on when an update r
| **field** | **type** | **length** | **description** | **key** |
| :------------- | :------- | :--------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------ |
| `id` | varchar | 255 | A repo's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<br/>For example, a Github repo's id is like "< github >:< GithubRepos >< GithubRepoId >". Eg. 'github:GithubRepos:384111310' | PK |
-| `language` | varchar | 255 | The language of repo.<br/>These are the [languages](https://api.github.com/repos/apache/incubator-devlake/languages) for apache/incubator-devlake | |
+| `language` | varchar | 255 | The language of repo.<br/>These are the [languages](https://api.github.com/repos/apache/incubator-devlake/languages) for apache/incubator-devlake | |
| `bytes` | int | | The byte counts for all files in those languages | |
| `created_date` | datetime | 3 | The field is filled in with the latest timestamp the query for a specific `repo_id` was done. | |
@@ -254,44 +251,60 @@ A ref is the abstraction of a branch or tag.
| **field** | **type** | **length** | **description** | **key** |
| :----------- | :------- | :--------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------- |
| `id` | varchar | 255 | A ref's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..."<br/>For example, a Github ref is composed of "github:GithubRepos:< GithubRepoId >:< RefUrl >". Eg. The id of release v5.3.0 of PingCAP/TiDB project is 'github:GithubRepos:384111310:refs/tags/v5.3.0' A repo's `id` is composed of "< plugin >:< Entity >:< PK0 >[:PK1]..." | PK |
-| `ref_name` | varchar | 255 | The name of ref. Eg. '[refs/tags/v0.9.3](https://github.com/apache/incubator-devlake/tree/v0.9.3)' | |
+| `ref_name` | varchar | 255 | The name of ref. Eg. '[refs/tags/v0.9.3](https://github.com/apache/incubator-devlake/tree/v0.9.3)' | |
| `repo_id` | varchar | 255 | The id of repo this ref belongs to | FK_repos.id |
| `commit_sha` | char | 40 | The commit this ref points to at the time of collection | |
| `is_default` | int | | <ul><li>0: the ref is the default branch. By the definition of [Github](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/changing-the-default-branch), the default branch is the base branch for pull requests and code commits.</li><li>1: not the default branch</li></ul> | |
| `merge_base` | char | 40 | The merge base commit of the main ref and the current ref | |
| `ref_type` | varchar | 64 | There're 2 typical types:<ul><li>BRANCH</li><li>TAG</li></ul> | |
-#### refs_commits_diffs
+#### commits_diffs
-This table shows the commits added in a new ref compared to an old ref. This table can be used to support tag-based analysis, for instance, '_No. of commits of a tag_', '_No. of merged pull request of a tag_', etc.
+This table shows the commits added in a new commit compared to an old commit. This table can be used to support tag-based and deploy-based metrics.
The records of this table are computed by [RefDiff](https://github.com/apache/incubator-devlake/tree/main/plugins/refdiff) plugin. The computation should be manually triggered after using [GitRepoExtractor](https://github.com/apache/incubator-devlake/tree/main/plugins/gitextractor) to collect commits and refs. The algorithm behind is similar to [this](https://github.com/apache/incubator-devlake/compare/v0.8.0%E2%80%A6v0.9.0).
-| **field** | **type** | **length** | **description** | **key** |
-| :------------------- | :------- | :--------- | :-------------------------------------------------------------- | :------------- |
-| `commit_sha` | char | 40 | One of the added commits in the new ref compared to the old ref | FK_commits.sha |
-| `new_ref_id` | varchar | 255 | The new ref's id for comparison | FK_refs.id |
-| `old_ref_id` | varchar | 255 | The old ref's id for comparison | FK_refs.id |
-| `new_ref_commit_sha` | char | 40 | The commit new ref points to at the time of collection | |
-| `old_ref_commit_sha` | char | 40 | The commit old ref points to at the time of collection | |
-| `sorting_index` | varchar | 255 | An index for debugging, please skip it | |
+| **field** | **type** | **length** | **description** | **key** |
+| :--------------- | :------- | :--------- | :------------------------------------------------------------------------- | :------ |
+| `new_commit_sha` | char | 40 | The commit new ref/deployment points to at the time of collection | PK |
+| `old_commit_sha` | char | 40 | The commit old ref/deployment points to at the time of collection | PK |
+| `commit_sha` | char | 40 | One of the added commits in the new ref compared to the old ref/deployment | PK |
+| `sorting_index` | varchar | 255 | An index for debugging, please skip it | |
+
+#### finished_commits_diffs
+
+This table shows the commits_diffs `new_commit_sha` and `old_commit_sha` pairs which are calculated successfully.
+
+| **field** | **type** | **length** | **description** | **key** |
+| :--------------- | :------- | :--------- | :-------------------------------------------------------------------- | :------ |
+| `new_commit_sha` | char | 40 | The new commit new ref/deployment points to at the time of collection | PK |
+| `old_commit_sha` | char | 40 | The commit old ref/deployment points to at the time of collection | PK |
+
+#### ref_commits
+
+| **field** | **type** | **length** | **description** | **key** |
+| :--------------- | :------- | :--------- | :----------------------------------------------------- | :------ |
+| `new_ref_id` | varchar | 255 | The new ref's id for comparison | PK |
+| `old_ref_id` | varchar | 255 | The old ref's id for comparison | PK |
+| `new_commit_sha` | char | 40 | The commit new ref points to at the time of collection | |
+| `old_commit_sha` | char | 40 | The commit old ref points to at the time of collection | |
#### commits
-| **field** | **type** | **length** | **description** | **key** |
-| :---------------- | :------- | :--------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------- |
-| `sha` | char | 40 | One of the added commits in the new ref compared to the old ref | FK_commits.sha |
-| `message` | varchar | 255 | Commit message | |
-| `author_name` | varchar | 255 | The value is set with command `git config user.name xxxxx` commit | |
-| `author_email` | varchar | 255 | The value is set with command `git config user.email xxxxx` author | |
-| `authored_date` | datetime | 3 | The date when this commit was originally made | |
-| `author_id` | varchar | 255 | The id of commit author | FK_accounts.id |
-| `committer_name` | varchar | 255 | The name of committer | |
-| `committer_email` | varchar | 255 | The email of committer | |
-| `committed_date` | datetime | 3 | The last time the commit gets modified.<br/>For example, when rebasing the branch where the commit is in on another branch, the committed_date changes. | |
-| `committer_id` | varchar | 255 | The id of committer | FK_accounts.id |
-| `additions` | int | | Added lines of code | |
-| `deletions` | int | | Deleted lines of code | |
+| **field** | **type** | **length** | **description** | **key** |
+| :---------------- | :------- | :--------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------- |
+| `sha` | char | 40 | One of the added commits in the new ref compared to the old ref | FK_commits.sha |
+| `message` | varchar | 255 | Commit message | |
+| `author_name` | varchar | 255 | The value is set with command `git config user.name xxxxx` commit | |
+| `author_email` | varchar | 255 | The value is set with command `git config user.email xxxxx` author | |
+| `authored_date` | datetime | 3 | The date when this commit was originally made | |
+| `author_id` | varchar | 255 | The id of commit author | FK_accounts.id |
+| `committer_name` | varchar | 255 | The name of committer | |
+| `committer_email` | varchar | 255 | The email of committer | |
+| `committed_date` | datetime | 3 | The last time the commit gets modified.<br/>For example, when rebasing the branch where the commit is in on another branch, the committed_date changes. | |
+| `committer_id` | varchar | 255 | The id of committer | FK_accounts.id |
+| `additions` | int | | Added lines of code | |
+| `deletions` | int | | Deleted lines of code | |
| `dev_eq` | int | | A metric that quantifies the amount of code contribution. The data can be retrieved from [AE plugin](https://github.com/apache/incubator-devlake/tree/main/plugins/ae). | |
#### commit_files
@@ -348,14 +361,14 @@ A pull request is the abstraction of GitHub pull request and Gitlab merge reques
| `title` | varchar | 255 | The title of pull request | |
| `description` | longtext | | The body/description of pull request | |
| `status` | varchar | 255 | the status of pull requests. For a Github pull request, the status can either be 'open' or 'closed'. | |
-| `parent_pr_id` | varchar | 255 | The id of the parent PR | |
-| `pull_request_key` | varchar | 255 | The key of PR. Eg, 1536 is the key of this [PR](https://github.com/apache/incubator-devlake/pull/1563) | |
+| `parent_pr_id` | varchar | 255 | The id of the parent PR | |
+| `pull_request_key` | varchar | 255 | The key of PR. Eg, 1536 is the key of this [PR](https://github.com/apache/incubator-devlake/pull/1563) | |
| `base_repo_id` | varchar | 255 | The repo that will be updated. | |
| `head_reop_id` | varchar | 255 | The repo containing the changes that will be added to the base. If the head repository is NULL, this means that the corresponding project had been deleted when DevLake processed the pull request. | |
| `base_ref` | varchar | 255 | The branch name in the base repo that will be updated | |
| `head_ref` | varchar | 255 | The branch name in the head repo that contains the changes that will be added to the base | |
-| `author_name` | varchar | 255 | The author's name of the pull request | |
-| `author_id` | varchar | 255 | The author's id of the pull request | |
+| `author_name` | varchar | 255 | The author's name of the pull request | |
+| `author_id` | varchar | 255 | The author's id of the pull request | |
| `url` | varchar | 255 | the web link of the pull request | |
| `type` | varchar | 255 | The work-type of a pull request. For example: feature-development, bug-fix, docs, etc.<br/>The value is transformed from Github pull request labels by configuring `GITHUB_PR_TYPE` in `.env` file during installation. | |
| `component` | varchar | 255 | The component this PR affects.<br/>The value is transformed from Github/Gitlab pull request labels by configuring `GITHUB_PR_COMPONENT` in `.env` file during installation. | |
@@ -363,14 +376,13 @@ A pull request is the abstraction of GitHub pull request and Gitlab merge reques
| `merged_date` | datetime | 3 | The time PR gets merged. Null when the PR is not merged. | |
| `closed_date` | datetime | 3 | The time PR closed. Null when the PR is not closed. | |
| `merge_commit_sha` | char | 40 | the merge commit of this PR. By the definition of [Github](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/changing-the-default-branch), when you click the default Merge pull request option on a pull request on Github, all commits from the feature branch are added to the base branch in a merge commit. | |
-| `base_commit_sha` | char | 40 | The base commit of this PR. | |
-| `head_commit_sha` | char | 40 | The head commit of this PR. | |
-| `coding_time_span`| bigint | | PR created_date - PR's first commit's authored_date | |
-| `review_lag` | bigint | | PR's first comment time - PR's created_date | |
-| `review_time_span`| bigint | | PR merged_date - PR's first comment time | |
-| `deploy_timespan` | bigint | | PR deployed date - PR merged_date | |
-| `change_timespan` | bigint | | PR cycle time, equals to coding_time_span + review_lag + review_time_span + deploy_timespan | |
-
+| `base_commit_sha` | char | 40 | The base commit of this PR. | |
+| `head_commit_sha` | char | 40 | The head commit of this PR. | |
+| `coding_time_span` | bigint | | PR created_date - PR's first commit's authored_date | |
+| `review_lag` | bigint | | PR's first comment time - PR's created_date | |
+| `review_time_span` | bigint | | PR merged_date - PR's first comment time | |
+| `deploy_timespan` | bigint | | PR deployed date - PR merged_date | |
+| `change_timespan` | bigint | | PR cycle time, equals to coding_time_span + review_lag + review_time_span + deploy_timespan | |
#### pull_request_labels
@@ -396,18 +408,17 @@ The list is additive. This means if a rebase with commit squashing takes place a
Normal comments, review bodies, reviews' inline comments of GitHub's pull requests or GitLab's merge requests.
-| **field** | **type** | **length** | **description** | **key** |
-| :---------------- | :------- | :--------- | :--------------------------------------------------------- | :------------------ |
-| `id` | varchar | 255 | Comment id | PK |
-| `pull_request_id` | varchar | 255 | Pull request id | FK_pull_requests.id |
-| `body` | longtext | | The body of the comments | |
-| `account_id` | varchar | 255 | The account who made the comment | FK_accounts.id |
-| `created_date` | datetime | 3 | Comment creation time | |
-| `position` | int | | Deprecated | |
-| `type` | varchar | 255 | - For normal comments: NORMAL<br/> - For review comments, ie. diff/inline comments: DIFF<br/> - For reviews' body (exist in GitHub but not GitLab): REVIEW | |
-| `review_id` | varchar | 255 | Review_id of the comment if the type is `REVIEW` or `DIFF` | |
-| `status` | varchar | 255 | Status of the comment | |
-
+| **field** | **type** | **length** | **description** | **key** |
+| :---------------- | :------- | :--------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------ |
+| `id` | varchar | 255 | Comment id | PK |
+| `pull_request_id` | varchar | 255 | Pull request id | FK_pull_requests.id |
+| `body` | longtext | | The body of the comments | |
+| `account_id` | varchar | 255 | The account who made the comment | FK_accounts.id |
+| `created_date` | datetime | 3 | Comment creation time | |
+| `position` | int | | Deprecated | |
+| `type` | varchar | 255 | - For normal comments: NORMAL<br/> - For review comments, ie. diff/inline comments: DIFF<br/> - For reviews' body (exist in GitHub but not GitLab): REVIEW | |
+| `review_id` | varchar | 255 | Review_id of the comment if the type is `REVIEW` or `DIFF` | |
+| `status` | varchar | 255 | Status of the comment | |
#### pull_request_events(WIP)
@@ -418,7 +429,7 @@ Events of pull requests.
| `id` | varchar | 255 | Event id | PK [...]
| `pull_request_id` | varchar | 255 | Pull request id | FK_ [...]
| `action` | varchar | 255 | The action to be taken, some values:<ul><li>`opened`: When the pull request has been opened</li><li>`closed`: When the pull request has been closed</li><li>`merged`: When Github detected that the pull request has been merged. No merges outside Github (i.e. Git based) are reported</li><li>`reoponed`: When a pull request is opened after being closed</li><li>`syncrhonize`: When new commits are added/removed to the head repository</li></ul> | [...]
-| `actor_id` | varchar | 255 | The account id of the event performer | [...]
+| `actor_id` | varchar | 255 | The account id of the event performer | FK_ [...]
| `created_date` | datetime | 3 | Event creation time | [...]
<br/>
@@ -429,46 +440,45 @@ Events of pull requests.
A cicd_pipeline is a series of builds that have connections or a standalone build.
-| **field** | **type** | **length** | **description** | **key** |
-| :-------- | :-------- | :-------- |:----------------------------------------------------------------------------------------------| :-------- |
-| `id` | varchar | 255 | This key is generated based on details from the original plugin | PK |
-| `name` | varchar | 255 | For gitlab, as there is no name for pipeline, so we use projectId, others have their own name | |
-| `result` | varchar | 100 | The result of this task | |
-| `status` | varchar | 100 | The status of this task | |
-| `type` | varchar | 100 | To indicate if this is a DEPLOYMENT | |
-| `duration_sec` | bigint unsigned | | how long does this task take | |
-| `started_date` | datetime | 3 | when did this task start | |
-| `finished_date` | datetime | 3 | when did this task finish | |
-| `environment` | varchar | 255 | To indicate the environment in which the task is running | |
+| **field** | **type** | **length** | **description** | **key** |
+| :-------------- | :-------------- | :--------- | :-------------------------------------------------------------------------------------------- | :------ |
+| `id` | varchar | 255 | This key is generated based on details from the original plugin | PK |
+| `name` | varchar | 255 | For gitlab, as there is no name for pipeline, so we use projectId, others have their own name | |
+| `result` | varchar | 100 | The result of this task | |
+| `status` | varchar | 100 | The status of this task | |
+| `type` | varchar | 100 | To indicate if this is a DEPLOYMENT | |
+| `duration_sec` | bigint unsigned | | how long does this task take | |
+| `started_date` | datetime | 3 | when did this task start | |
+| `finished_date` | datetime | 3 | when did this task finish | |
+| `environment` | varchar | 255 | To indicate the environment in which the task is running | |
#### cicd_pipeline_commits
-| **field** | **type** | **length** | **description** | **key** |
-| :-------- | :-------- | :-------- |:----------------------------------------------------------------| :-------- |
-| `pipeline_id` | varchar | 255 | This key is generated based on details from the original plugin | PK |
-| `commit_sha` | varchar | 255 | The commit that trigger this pipeline | PK |
-| `branch` | varchar | 255 | The branch that trigger this pipeline | |
-| `repo` | varchar | 255 | | |
-| `repo_id` | varchar | 255 | The repo that this pipeline belongs to | |
-| `repo_url` | longtext | | | |
+| **field** | **type** | **length** | **description** | **key** |
+| :------------ | :------- | :--------- | :-------------------------------------------------------------- | :------ |
+| `pipeline_id` | varchar | 255 | This key is generated based on details from the original plugin | PK |
+| `commit_sha` | varchar | 255 | The commit that trigger this pipeline | PK |
+| `branch` | varchar | 255 | The branch that trigger this pipeline | |
+| `repo` | varchar | 255 | | |
+| `repo_id` | varchar | 255 | The repo that this pipeline belongs to | |
+| `repo_url` | longtext | | | |
#### cicd_tasks
A cicd_task is a single job of ci/cd.
-| **field** | **type** | **length** | **description** | **key** |
-| :-------- | :-------- |:-------------------------------|:----------------------------------------------------------------| :-------- |
-| `id` | varchar | 255 | This key is generated based on details from the original plugin | PK |
-| `name` | varchar | 255 | | |
-| `pipeline_id` | varchar | 255 | The id of pipeline | |
-| `result` | varchar | 100 | The result of this task | |
-| `status` | varchar | 100 | The status of this task | |
-| `type` | varchar | 100 | To indicate if this is a DEPLOYMENT | |
-| `duration_sec` | bigint unsigned | | how long does this task take | |
-| `started_date` | datetime | 3 | when did this task start | |
-| `finished_date` | datetime | 3 | when did this task finish | |
-| `environment` | varchar | 255 | To indicate the environment in which the task is running | |
-
+| **field** | **type** | **length** | **description** | **key** |
+| :-------------- | :-------------- | :--------- | :-------------------------------------------------------------- | :------ |
+| `id` | varchar | 255 | This key is generated based on details from the original plugin | PK |
+| `name` | varchar | 255 | | |
+| `pipeline_id` | varchar | 255 | The id of pipeline | |
+| `result` | varchar | 100 | The result of this task | |
+| `status` | varchar | 100 | The status of this task | |
+| `type` | varchar | 100 | To indicate if this is a DEPLOYMENT | |
+| `duration_sec` | bigint unsigned | | how long does this task take | |
+| `started_date` | datetime | 3 | when did this task start | |
+| `finished_date` | datetime | 3 | when did this task finish | |
+| `environment` | varchar | 255 | To indicate the environment in which the task is running | |
### Cross-Domain Entities
@@ -495,12 +505,12 @@ This table shows the issues closed by pull requests. It's a medium-level mapping
The data is extracted from the body of pull requests conforming to certain regular expression. The regular expression can be defined in GITHUB_PR_BODY_CLOSE_PATTERN in the .env file
-| **field** | **type** | **length** | **description** | **key** |
-| :-------------------- | :------- | :--------- | :------------------ | :------------------ |
-| `pull_request_id` | char | 40 | Pull request id | FK_pull_requests.id |
-| `issue_id` | varchar | 255 | Issue id | FK_issues.id |
-| `pull_request_number` | varchar | 255 | Pull request key | |
-| `issue_number` | varchar | 255 | Issue key | |
+| **field** | **type** | **length** | **description** | **key** |
+| :-------------------- | :------- | :--------- | :--------------- | :------------------ |
+| `pull_request_id` | char | 40 | Pull request id | FK_pull_requests.id |
+| `issue_id` | varchar | 255 | Issue id | FK_issues.id |
+| `pull_request_number` | varchar | 255 | Pull request key | |
+| `issue_number` | varchar | 255 | Issue key | |
#### board_repos (Deprecated)
@@ -514,10 +524,10 @@ A way to link "issue tracking" and "source code management" domain by mapping `b
#### accounts
This table stores of user accounts across different tools such as GitHub, Jira, GitLab, etc. This table can be joined to get the metadata of all accounts.
- metrics, such as _'No. of Issue closed by contributor', 'No. of commits by contributor',_
+metrics, such as _'No. of Issue closed by contributor', 'No. of commits by contributor',_
| **field** | **type** | **length** | **description** | **key** |
-| :------------- | :------- | :--------- |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :------ |
+| :------------- | :------- | :--------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------ |
| `id` | varchar | 255 | An account's `id` is the identifier of the account of a specific tool. It is composed of "< Plugin >:< Entity >:< PK0 >[:PK1]..."<br/>For example, a Github account's id is composed of "< github >:< GithubAccounts >:< GithubUserId >)". Eg. 'github:GithubUsers:14050754' | PK |
| `email` | varchar | 255 | Email of the account | |
| `full_name` | varchar | 255 | Full name | |
@@ -528,6 +538,7 @@ This table stores of user accounts across different tools such as GitHub, Jira,
| `status` | int | | 0: default, the user is active. 1: the user is not active | |
#### users
+
| **field** | **type** | **length** | **description** | **key** |
| --------- | -------- | ---------- | ----------------------------- | ------- |
| `id` | varchar | 255 | id of a person | PK |
@@ -535,12 +546,14 @@ This table stores of user accounts across different tools such as GitHub, Jira,
| `name` | varchar | 255 | name of a person | |
#### user_accounts
+
| **field** | **type** | **length** | **description** | **key** |
| ------------ | -------- | ---------- | --------------- | ---------------- |
| `user_id` | varchar | 255 | users.id | Composite PK, FK |
| `account_id` | varchar | 255 | accounts.id | Composite PK, FK |
#### teams
+
| **field** | **type** | **length** | **description** | **key** |
| --------------- | -------- | ---------- | -------------------------------------------------- | ------- |
| `id` | varchar | 255 | id from the data sources, decided by DevLake users | PK |
@@ -550,22 +563,21 @@ This table stores of user accounts across different tools such as GitHub, Jira,
| `sorting_index` | int | 255 | the field to sort team | |
#### team_users
+
| **field** | **type** | **length** | **description** | **key** |
| --------- | -------- | ---------- | ----------------------------------------------- | ---------------- |
| `team_id` | varchar | 255 | Full name of the team. Eg. team A, team B, etc. | Composite PK, FK |
| `user_id` | varchar | 255 | users.id | Composite PK, FK |
-
<br/>
## DWM Entities - (Data Warehouse Middle)
DWM entities are the slight aggregation and operation of DWD to store more organized details or middle-level metrics.
-
#### refs_issues_diffs
-This table shows the issues fixed by commits added in a new ref compared to an old one. The data is computed from [table.ref_commits_diff](#refs_commits_diffs), [table.pull_requests](#pull_requests), [table.pull_request_commits](#pull_request_commits), and [table.pull_request_issues](#pull_request_issues).
+This table shows the issues fixed by commits added in a new ref compared to an old one. The data is computed from [table.commits_diffs](#commits_diffs), [table.finished_commits_diffs](#finished_commits_diffs), [table.pull_requests](#pull_requests), [table.pull_request_commits](#pull_request_commits), and [table.pull_request_issues](#pull_request_issues).
This table can support tag-based analysis, for instance, '_No. of bugs closed in a tag_'.
@@ -578,7 +590,6 @@ This table can support tag-based analysis, for instance, '_No. of bugs closed in
| `issue_number` | varchar | 255 | Issue number | |
| `issue_id` | varchar | 255 | Issue id | FK_issues.id |
-
## Get Domain Layer Models in Developer Mode
When developing a new plugin, you need to refer to domain layer models, as all raw data should be transformed to domain layer data to provide standardized metrics across tools. Please use the following method to access the domain data models.
@@ -588,8 +599,8 @@ import "github.com/apache/incubator-devlake/models/domainlayer/domaininfo"
domaininfo := domaininfo.GetDomainTablesInfo()
for _, table := range domaininfo {
- // do something
+ // do something
}
```
-If you want to learn more about plugin models, please visit [PluginImplementation](https://devlake.apache.org/docs/DeveloperManuals/PluginImplementation)
\ No newline at end of file
+If you want to learn more about plugin models, please visit [PluginImplementation](https://devlake.apache.org/docs/DeveloperManuals/PluginImplementation)
diff --git a/docs/Plugins/refdiff.md b/docs/Plugins/refdiff.md
index 4ba983c16..53ec882c2 100644
--- a/docs/Plugins/refdiff.md
+++ b/docs/Plugins/refdiff.md
@@ -6,7 +6,12 @@ description: >
## Summary
-For development workload analysis, we often need to know how many commits have been created between 2 releases. This plugin calculates which commits differ between 2 commits, and the commits are produced by Ref (branch/tag) or by deployments, and the result will be stored back into the database for further analysis.
+RefDiff is a plugin that performs calculation tasks and has 2 main purposes.
+
+- Calculate the difference in commits between releases/tags to [analyze the amount of code in each release](add an anchor link)
+- Calculate the difference in commits between deployments to [calculate DORA metrics](add an anchor link)
+
+And the output of RefDiff is stored in the table commits_diffs, finished_commits_diffs, ref_commits.
## Important Note
diff --git a/static/img/DomainLayerSchema/domain_layer_model_v0.14.ndm2 b/static/img/DomainLayerSchema/domain_layer_model_v0.14.ndm2
index d331b4944..197f3d1b9 100644
--- a/static/img/DomainLayerSchema/domain_layer_model_v0.14.ndm2
+++ b/static/img/DomainLayerSchema/domain_layer_model_v0.14.ndm2
@@ -3303,158 +3303,6 @@
"triggers": [],
"tablePartitions": []
},
- {
- "objectType": "Table_MYSQL",
- "name": "issue_commits",
- "comment": "",
- "engine": "",
- "characterSet": "",
- "collation": "",
- "autoIncrement": 0,
- "tablespace": "",
- "storage": "",
- "insertMethod": "",
- "connection": "",
- "checksum": false,
- "rowFormat": "",
- "avgRowLength": 0,
- "maxRows": 0,
- "minRows": 0,
- "keyBlockSize": 0,
- "packKeys": "",
- "delayKeyWrite": false,
- "dataDirectory": "",
- "indexDirectory": "",
- "statsAutoRecalc": "",
- "statsPersistent": "",
- "statsSamplePages": 0,
- "union": "",
- "pageCheckSum": false,
- "transactional": false,
- "compression": "",
- "oldName": "",
- "encryption": false,
- "createOptions": "",
- "createTime": "",
- "checkTime": "",
- "dataFree": 0,
- "dataLength": 0,
- "indexLength": 0,
- "maxDataLength": 0,
- "rows": 0,
- "updateTime": "",
- "DDL": "",
- "partitionBy": "",
- "partitionByExpr": "",
- "partitions": 0,
- "partitionKeyAlgorithm": "",
- "subPartitionBy": "",
- "subPartitionByExpr": "",
- "subPartitions": 0,
- "subPartitionKeyAlgorithm": "",
- "fields": [
- {
- "objectType": "TableField_MYSQL",
- "name": "issue_id",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "issue_id"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "commit_sha",
- "type": "char",
- "length": 40,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "commit_sha"
- }
- ],
- "indexes": [],
- "primaryKey": {
- "objectType": "PrimaryKey_MYSQL",
- "name": "",
- "fields": [],
- "oldName": "",
- "indexMethod": "",
- "comment": ""
- },
- "foreignKeys": [
- {
- "objectType": "ForeignKey_MYSQL",
- "name": "fk_board_issues_issues_1_copy_1",
- "fields": [
- "issue_origin_key"
- ],
- "referenceSchema": "Default",
- "referenceTable": "issues",
- "referenceFields": [
- "origin_key"
- ],
- "onDelete": "",
- "onUpdate": "",
- "sourceCardinality": "NoneRelationship",
- "targetCardinality": "NoneRelationship",
- "oldName": ""
- },
- {
- "objectType": "ForeignKey_MYSQL",
- "name": "fk_commit_issues_commits_1",
- "fields": [
- "commit_origin_key"
- ],
- "referenceSchema": "Default",
- "referenceTable": "commits",
- "referenceFields": [
- "origin_key"
- ],
- "onDelete": "",
- "onUpdate": "",
- "sourceCardinality": "NoneRelationship",
- "targetCardinality": "NoneRelationship",
- "oldName": ""
- }
- ],
- "checks": [],
- "triggers": [],
- "tablePartitions": []
- },
{
"objectType": "Table_MYSQL",
"name": "refs_issues_diffs",
@@ -5757,665 +5605,7 @@
"issue_origin_key"
],
"referenceSchema": "Default",
- "referenceTable": "issues",
- "referenceFields": [
- "origin_key"
- ],
- "onDelete": "",
- "onUpdate": "",
- "sourceCardinality": "NoneRelationship",
- "targetCardinality": "NoneRelationship",
- "oldName": ""
- }
- ],
- "checks": [],
- "triggers": [],
- "tablePartitions": []
- },
- {
- "objectType": "Table_MYSQL",
- "name": "issue_comments(WIP)",
- "comment": "",
- "engine": "",
- "characterSet": "",
- "collation": "",
- "autoIncrement": 0,
- "tablespace": "",
- "storage": "",
- "insertMethod": "",
- "connection": "",
- "checksum": false,
- "rowFormat": "",
- "avgRowLength": 0,
- "maxRows": 0,
- "minRows": 0,
- "keyBlockSize": 0,
- "packKeys": "",
- "delayKeyWrite": false,
- "dataDirectory": "",
- "indexDirectory": "",
- "statsAutoRecalc": "",
- "statsPersistent": "",
- "statsSamplePages": 0,
- "union": "",
- "pageCheckSum": false,
- "transactional": false,
- "compression": "",
- "oldName": "",
- "encryption": false,
- "createOptions": "",
- "createTime": "",
- "checkTime": "",
- "dataFree": 0,
- "dataLength": 0,
- "indexLength": 0,
- "maxDataLength": 0,
- "rows": 0,
- "updateTime": "",
- "DDL": "",
- "partitionBy": "",
- "partitionByExpr": "",
- "partitions": 0,
- "partitionKeyAlgorithm": "",
- "subPartitionBy": "",
- "subPartitionByExpr": "",
- "subPartitions": 0,
- "subPartitionKeyAlgorithm": "",
- "fields": [
- {
- "objectType": "TableField_MYSQL",
- "name": "id",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": false,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "id"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "issue_id",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "issue_id"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "account_id",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "account_id"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "body",
- "type": "longtext",
- "length": -2147483648,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "body"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "created_date",
- "type": "datetime",
- "length": 3,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "created_date"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "updated_date",
- "type": "datetime",
- "length": 3,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "updated_date"
- }
- ],
- "indexes": [],
- "primaryKey": {
- "objectType": "PrimaryKey_MYSQL",
- "name": "",
- "fields": [
- {
- "objectType": "IndexField_MYSQL",
- "name": "id",
- "keyLength": 0,
- "order": "",
- "oldName": ""
- }
- ],
- "oldName": "",
- "indexMethod": "",
- "comment": ""
- },
- "foreignKeys": [
- {
- "objectType": "ForeignKey_MYSQL",
- "name": "fk_issue_comments_issues_1",
- "fields": [
- "issue_origin_key"
- ],
- "referenceSchema": "Default",
- "referenceTable": "issues",
- "referenceFields": [
- "origin_key"
- ],
- "onDelete": "",
- "onUpdate": "",
- "sourceCardinality": "NoneRelationship",
- "targetCardinality": "NoneRelationship",
- "oldName": ""
- },
- {
- "objectType": "ForeignKey_MYSQL",
- "name": "fk_issue_comments_users_1",
- "fields": [
- "user_origin_key"
- ],
- "referenceSchema": "Default",
- "referenceTable": "accounts",
- "referenceFields": [
- "origin_key"
- ],
- "onDelete": "",
- "onUpdate": "",
- "sourceCardinality": "NoneRelationship",
- "targetCardinality": "NoneRelationship",
- "oldName": ""
- }
- ],
- "checks": [],
- "triggers": [],
- "tablePartitions": []
- },
- {
- "objectType": "Table_MYSQL",
- "name": "issue_changelogs",
- "comment": "",
- "engine": "",
- "characterSet": "",
- "collation": "",
- "autoIncrement": 0,
- "tablespace": "",
- "storage": "",
- "insertMethod": "",
- "connection": "",
- "checksum": false,
- "rowFormat": "",
- "avgRowLength": 0,
- "maxRows": 0,
- "minRows": 0,
- "keyBlockSize": 0,
- "packKeys": "",
- "delayKeyWrite": false,
- "dataDirectory": "",
- "indexDirectory": "",
- "statsAutoRecalc": "",
- "statsPersistent": "",
- "statsSamplePages": 0,
- "union": "",
- "pageCheckSum": false,
- "transactional": false,
- "compression": "",
- "oldName": "",
- "encryption": false,
- "createOptions": "",
- "createTime": "",
- "checkTime": "",
- "dataFree": 0,
- "dataLength": 0,
- "indexLength": 0,
- "maxDataLength": 0,
- "rows": 0,
- "updateTime": "",
- "DDL": "",
- "partitionBy": "",
- "partitionByExpr": "",
- "partitions": 0,
- "partitionKeyAlgorithm": "",
- "subPartitionBy": "",
- "subPartitionByExpr": "",
- "subPartitions": 0,
- "subPartitionKeyAlgorithm": "",
- "fields": [
- {
- "objectType": "TableField_MYSQL",
- "name": "id",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": false,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "id"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "issue_id",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "issue_id"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "author_id",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "author_id"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "author_name",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "author_name"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "field_id",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "field_id"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "field_name",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "field_name"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "original_from_value",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "original_from_value"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "original_to_value",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "original_to_value"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "from_value",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "from_value"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "to_value",
- "type": "varchar",
- "length": 255,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "to_value"
- },
- {
- "objectType": "TableField_MYSQL",
- "name": "created_date",
- "type": "datetime",
- "length": 3,
- "decimals": 0,
- "isUnsigned": false,
- "isZeroFill": false,
- "setEnumValues": "",
- "isBinary": false,
- "charset": "",
- "collation": "",
- "isNullable": true,
- "defaultType": "Others",
- "defaultValue": "",
- "isOnUpdateCurrentTimestamp": false,
- "isAutoInc": false,
- "comment": "",
- "columnFormat": "",
- "storage": "",
- "isVirtual": false,
- "isGeneratedAlways": false,
- "virtualExpr": "",
- "virtualType": "",
- "oldName": "created_date"
- }
- ],
- "indexes": [],
- "primaryKey": {
- "objectType": "PrimaryKey_MYSQL",
- "name": "",
- "fields": [
- {
- "objectType": "IndexField_MYSQL",
- "name": "id",
- "keyLength": 0,
- "order": "",
- "oldName": ""
- }
- ],
- "oldName": "",
- "indexMethod": "",
- "comment": ""
- },
- "foreignKeys": [
- {
- "objectType": "ForeignKey_MYSQL",
- "name": "fk_issue_changelog_issues_1",
- "fields": [
- "issue_origin_key"
- ],
- "referenceSchema": "Default",
- "referenceTable": "issues",
- "referenceFields": [
- "origin_key"
- ],
- "onDelete": "",
- "onUpdate": "",
- "sourceCardinality": "NoneRelationship",
- "targetCardinality": "NoneRelationship",
- "oldName": ""
- },
- {
- "objectType": "ForeignKey_MYSQL",
- "name": "fk_issue_changelog_users_1",
- "fields": [
- "actor_origin_key"
- ],
- "referenceSchema": "Default",
- "referenceTable": "accounts",
+ "referenceTable": "issues",
"referenceFields": [
"origin_key"
],
@@ -10827,6 +10017,225 @@
"triggers": [],
"tablePartitions": []
},
+ {
+ "objectType": "Table_MYSQL",
+ "name": "ref_commits",
+ "comment": "",
+ "engine": "",
+ "characterSet": "",
+ "collation": "",
+ "autoIncrement": 0,
+ "tablespace": "",
+ "storage": "",
+ "insertMethod": "",
+ "connection": "",
+ "checksum": false,
+ "rowFormat": "",
+ "avgRowLength": 0,
+ "maxRows": 0,
+ "minRows": 0,
+ "keyBlockSize": 0,
+ "packKeys": "",
+ "delayKeyWrite": false,
+ "dataDirectory": "",
+ "indexDirectory": "",
+ "statsAutoRecalc": "",
+ "statsPersistent": "",
+ "statsSamplePages": 0,
+ "union": "",
+ "pageCheckSum": false,
+ "transactional": false,
+ "compression": "",
+ "oldName": "",
+ "encryption": false,
+ "createOptions": "",
+ "createTime": "",
+ "checkTime": "",
+ "dataFree": 0,
+ "dataLength": 0,
+ "indexLength": 0,
+ "maxDataLength": 0,
+ "rows": 0,
+ "updateTime": "",
+ "DDL": "",
+ "partitionBy": "",
+ "partitionByExpr": "",
+ "partitions": 0,
+ "partitionKeyAlgorithm": "",
+ "subPartitionBy": "",
+ "subPartitionByExpr": "",
+ "subPartitions": 0,
+ "subPartitionKeyAlgorithm": "",
+ "fields": [
+ {
+ "objectType": "TableField_MYSQL",
+ "name": "new_commit_sha",
+ "type": "char",
+ "length": 40,
+ "decimals": 0,
+ "isUnsigned": false,
+ "isZeroFill": false,
+ "setEnumValues": "",
+ "isBinary": false,
+ "charset": "",
+ "collation": "",
+ "isNullable": false,
+ "defaultType": "Others",
+ "defaultValue": "",
+ "isOnUpdateCurrentTimestamp": false,
+ "isAutoInc": false,
+ "comment": "",
+ "columnFormat": "",
+ "storage": "",
+ "isVirtual": false,
+ "isGeneratedAlways": false,
+ "virtualExpr": "",
+ "virtualType": "",
+ "oldName": "new_commit_sha"
+ },
+ {
+ "objectType": "TableField_MYSQL",
+ "name": "old_commit_sha",
+ "type": "char",
+ "length": 40,
+ "decimals": 0,
+ "isUnsigned": false,
+ "isZeroFill": false,
+ "setEnumValues": "",
+ "isBinary": false,
+ "charset": "",
+ "collation": "",
+ "isNullable": false,
+ "defaultType": "Others",
+ "defaultValue": "",
+ "isOnUpdateCurrentTimestamp": false,
+ "isAutoInc": false,
+ "comment": "",
+ "columnFormat": "",
+ "storage": "",
+ "isVirtual": false,
+ "isGeneratedAlways": false,
+ "virtualExpr": "",
+ "virtualType": "",
+ "oldName": "old_commit_sha"
+ },
+ {
+ "objectType": "TableField_MYSQL",
+ "name": "new_ref_id",
+ "type": "char",
+ "length": 255,
+ "decimals": -2147483648,
+ "isUnsigned": false,
+ "isZeroFill": false,
+ "setEnumValues": "",
+ "isBinary": false,
+ "charset": "",
+ "collation": "",
+ "isNullable": false,
+ "defaultType": "Others",
+ "defaultValue": "",
+ "isOnUpdateCurrentTimestamp": false,
+ "isAutoInc": false,
+ "comment": "",
+ "columnFormat": "",
+ "storage": "",
+ "isVirtual": false,
+ "isGeneratedAlways": false,
+ "virtualExpr": "",
+ "virtualType": "",
+ "oldName": "new_ref_id"
+ },
+ {
+ "objectType": "TableField_MYSQL",
+ "name": "olde_red_id",
+ "type": "char",
+ "length": 255,
+ "decimals": -2147483648,
+ "isUnsigned": false,
+ "isZeroFill": false,
+ "setEnumValues": "",
+ "isBinary": false,
+ "charset": "",
+ "collation": "",
+ "isNullable": false,
+ "defaultType": "Others",
+ "defaultValue": "",
+ "isOnUpdateCurrentTimestamp": false,
+ "isAutoInc": false,
+ "comment": "",
+ "columnFormat": "",
+ "storage": "",
+ "isVirtual": false,
+ "isGeneratedAlways": false,
+ "virtualExpr": "",
+ "virtualType": "",
+ "oldName": "olde_red_id"
+ }
+ ],
+ "indexes": [],
+ "primaryKey": {
+ "objectType": "PrimaryKey_MYSQL",
+ "name": "",
+ "fields": [
+ {
+ "objectType": "IndexField_MYSQL",
+ "name": "new_ref_id",
+ "keyLength": 0,
+ "order": "",
+ "oldName": ""
+ },
+ {
+ "objectType": "IndexField_MYSQL",
+ "name": "olde_red_id",
+ "keyLength": 0,
+ "order": "",
+ "oldName": ""
+ }
+ ],
+ "oldName": "",
+ "indexMethod": "",
+ "comment": ""
+ },
+ "foreignKeys": [
+ {
+ "objectType": "ForeignKey_MYSQL",
+ "name": "fk_ref_commits_ref_commits_1",
+ "fields": [
+ "new_commit_sha"
+ ],
+ "referenceSchema": "Default",
+ "referenceTable": "finished_commits_diffs",
+ "referenceFields": [
+ "new_commit_sha"
+ ],
+ "onDelete": "",
+ "onUpdate": "",
+ "sourceCardinality": "NoneRelationship",
+ "targetCardinality": "NoneRelationship",
+ "oldName": ""
+ },
+ {
+ "objectType": "ForeignKey_MYSQL",
+ "name": "fk_ref_commits_ref_commits_2",
+ "fields": [
+ "old_commit_sha"
+ ],
+ "referenceSchema": "Default",
+ "referenceTable": "finished_commits_diffs",
+ "referenceFields": [
+ "old_commit_sha"
+ ],
+ "onDelete": "",
+ "onUpdate": "",
+ "sourceCardinality": "NoneRelationship",
+ "targetCardinality": "NoneRelationship",
+ "oldName": ""
+ }
+ ],
+ "checks": [],
+ "triggers": [],
+ "tablePartitions": []
+ },
{
"objectType": "Table_MYSQL",
"name": "finished_commits_diffs",
@@ -12495,22 +11904,6 @@
"isFontBold": false,
"isFontItalic": false
},
- {
- "type": "table",
- "schemaName": "Default",
- "tableName": "issue_commits",
- "x": 600,
- "y": 1144,
- "width": 200,
- "height": 89,
- "isBold": false,
- "titleColor": {
- "r": 92,
- "g": 240,
- "b": 158,
- "a": 1
- }
- },
{
"type": "table",
"schemaName": "Default",
@@ -12583,43 +11976,11 @@
"y": 1515,
"width": 200,
"height": 88,
- "isBold": false,
- "titleColor": {
- "r": 55,
- "g": 131,
- "b": 192,
- "a": 1
- }
- },
- {
- "type": "table",
- "schemaName": "Default",
- "tableName": "issue_comments(WIP)",
- "x": 105,
- "y": 1640,
- "width": 200,
- "height": 171,
- "isBold": false,
- "titleColor": {
- "r": 253,
- "g": 181,
- "b": 12,
- "a": 1
- }
- },
- {
- "type": "table",
- "schemaName": "Default",
- "tableName": "issue_changelogs",
- "x": 104,
- "y": 1820,
- "width": 200,
- "height": 291,
- "isBold": false,
- "titleColor": {
- "r": 253,
- "g": 181,
- "b": 12,
+ "isBold": false,
+ "titleColor": {
+ "r": 55,
+ "g": 131,
+ "b": 192,
"a": 1
}
},
@@ -12960,7 +12321,7 @@
"schemaName": "Default",
"tableName": "commits_diffs",
"x": 976,
- "y": 1154,
+ "y": 1169,
"width": 200,
"height": 172,
"isBold": false,
@@ -13229,8 +12590,8 @@
"type": "table",
"schemaName": "Default",
"tableName": "finished_commits_diffs",
- "x": 979,
- "y": 1366,
+ "x": 981,
+ "y": 1399,
"width": 197,
"height": 94,
"isBold": false,
@@ -13240,6 +12601,22 @@
"b": 192,
"a": 1
}
+ },
+ {
+ "type": "table",
+ "schemaName": "Default",
+ "tableName": "ref_commits",
+ "x": 671,
+ "y": 1404,
+ "width": 197,
+ "height": 138,
+ "isBold": false,
+ "titleColor": {
+ "r": 55,
+ "g": 131,
+ "b": 192,
+ "a": 1
+ }
}
],
"layers": [],
@@ -13784,86 +13161,6 @@
"isVisible": false
}
},
- {
- "name": "fk_board_issues_issues_1_copy_1",
- "sourceTableName": "issue_commits",
- "sourceSchemaName": "Default",
- "lineWidth": 1,
- "visible": true,
- "vertices": [
- {
- "x": 585,
- "y": 1191
- },
- {
- "x": 326,
- "y": 1191
- },
- {
- "x": 326,
- "y": 1095
- },
- {
- "x": 321,
- "y": 1095
- }
- ],
- "label": {
- "x": 373,
- "y": 1152,
- "width": 222,
- "height": 31,
- "fontName": "Arial Unicode MS",
- "fontSize": 14,
- "fontColor": {
- "r": 51,
- "g": 51,
- "b": 51,
- "a": 1
- },
- "isFontBold": false,
- "isFontItalic": false,
- "isVisible": false
- }
- },
- {
- "name": "fk_commit_issues_commits_1",
- "sourceTableName": "issue_commits",
- "sourceSchemaName": "Default",
- "lineWidth": 1,
- "visible": true,
- "vertices": [
- {
- "x": 815,
- "y": 1211
- },
- {
- "x": 846,
- "y": 1242
- },
- {
- "x": 1236,
- "y": 1242
- }
- ],
- "label": {
- "x": 805,
- "y": 1172,
- "width": 192,
- "height": 31,
- "fontName": "Arial Unicode MS",
- "fontSize": 14,
- "fontColor": {
- "r": 51,
- "g": 51,
- "b": 51,
- "a": 1
- },
- "isFontBold": false,
- "isFontItalic": false,
- "isVisible": false
- }
- },
{
"name": "fk_repo_commits_repos_1",
"sourceTableName": "repo_commits",
@@ -14126,170 +13423,6 @@
"isVisible": false
}
},
- {
- "name": "fk_issue_comments_issues_1",
- "sourceTableName": "issue_comments(WIP)",
- "sourceSchemaName": "Default",
- "lineWidth": 1,
- "visible": true,
- "vertices": [
- {
- "x": 90,
- "y": 1685
- },
- {
- "x": 77,
- "y": 1673
- },
- {
- "x": 77,
- "y": 1091
- },
- {
- "x": 91,
- "y": 1091
- }
- ],
- "label": {
- "x": 106,
- "y": 1651,
- "width": 192,
- "height": 31,
- "fontName": "Arial Unicode MS",
- "fontSize": 14,
- "fontColor": {
- "r": 51,
- "g": 51,
- "b": 51,
- "a": 1
- },
- "isFontBold": false,
- "isFontItalic": false,
- "isVisible": false
- }
- },
- {
- "name": "fk_issue_comments_users_1",
- "sourceTableName": "issue_comments(WIP)",
- "sourceSchemaName": "Default",
- "lineWidth": 1,
- "visible": true,
- "vertices": [
- {
- "x": 320,
- "y": 1706
- },
- {
- "x": 652,
- "y": 1694
- },
- {
- "x": 652,
- "y": 1648
- },
- {
- "x": 658,
- "y": 1648
- }
- ],
- "label": {
- "x": 310,
- "y": 1667,
- "width": 187,
- "height": 31,
- "fontName": "Arial Unicode MS",
- "fontSize": 14,
- "fontColor": {
- "r": 51,
- "g": 51,
- "b": 51,
- "a": 1
- },
- "isFontBold": false,
- "isFontItalic": false,
- "isVisible": false
- }
- },
- {
- "name": "fk_issue_changelog_issues_1",
- "sourceTableName": "issue_changelogs",
- "sourceSchemaName": "Default",
- "lineWidth": 1,
- "visible": true,
- "vertices": [
- {
- "x": 89,
- "y": 1887
- },
- {
- "x": 70,
- "y": 1872
- },
- {
- "x": 70,
- "y": 1087
- },
- {
- "x": 91,
- "y": 1087
- }
- ],
- "label": {
- "x": 105,
- "y": 1858,
- "width": 192,
- "height": 31,
- "fontName": "Arial Unicode MS",
- "fontSize": 14,
- "fontColor": {
- "r": 51,
- "g": 51,
- "b": 51,
- "a": 1
- },
- "isFontBold": false,
- "isFontItalic": false,
- "isVisible": false
- }
- },
- {
- "name": "fk_issue_changelog_users_1",
- "sourceTableName": "issue_changelogs",
- "sourceSchemaName": "Default",
- "lineWidth": 1,
- "visible": true,
- "vertices": [
- {
- "x": 319,
- "y": 1908
- },
- {
- "x": 658,
- "y": 1893
- },
- {
- "x": 658,
- "y": 1655
- }
- ],
- "label": {
- "x": 309,
- "y": 1869,
- "width": 187,
- "height": 31,
- "fontName": "Arial Unicode MS",
- "fontSize": 14,
- "fontColor": {
- "r": 51,
- "g": 51,
- "b": 51,
- "a": 1
- },
- "isFontBold": false,
- "isFontItalic": false,
- "isVisible": false
- }
- },
{
"name": "fk_repos_users_1",
"sourceTableName": "repos",
@@ -14831,7 +13964,7 @@
"vertices": [
{
"x": 1191,
- "y": 1201
+ "y": 1216
},
{
"x": 1191,
@@ -14844,7 +13977,7 @@
],
"label": {
"x": 1181,
- "y": 1162,
+ "y": 1177,
"width": 149,
"height": 31,
"fontName": "Arial Unicode MS",
@@ -14869,7 +14002,7 @@
"vertices": [
{
"x": 961,
- "y": 1199
+ "y": 1214
},
{
"x": 962,
@@ -14878,7 +14011,7 @@
],
"label": {
"x": 977,
- "y": 1170,
+ "y": 1185,
"width": 187,
"height": 31,
"fontName": "Arial Unicode MS",
@@ -15528,17 +14661,17 @@
"visible": true,
"vertices": [
{
- "x": 964,
- "y": 1431
+ "x": 966,
+ "y": 1464
},
{
"x": 961,
- "y": 1226
+ "y": 1241
}
],
"label": {
- "x": 980,
- "y": 1402,
+ "x": 982,
+ "y": 1435,
"width": 310,
"height": 31,
"fontName": "Arial Unicode MS",
@@ -15562,17 +14695,17 @@
"visible": true,
"vertices": [
{
- "x": 1191,
- "y": 1410
+ "x": 1193,
+ "y": 1443
},
{
"x": 1191,
- "y": 1247
+ "y": 1262
}
],
"label": {
- "x": 1187,
- "y": 1381,
+ "x": 1189,
+ "y": 1414,
"width": 310,
"height": 31,
"fontName": "Arial Unicode MS",
@@ -15596,8 +14729,8 @@
"visible": true,
"vertices": [
{
- "x": 1074,
- "y": 1475
+ "x": 1076,
+ "y": 1508
},
{
"x": 1076,
@@ -15613,8 +14746,8 @@
}
],
"label": {
- "x": 1076,
- "y": 1424,
+ "x": 1078,
+ "y": 1457,
"width": 310,
"height": 31,
"fontName": "Arial Unicode MS",
@@ -15629,6 +14762,74 @@
"isFontItalic": false,
"isVisible": false
}
+ },
+ {
+ "name": "fk_ref_commits_ref_commits_1",
+ "sourceTableName": "ref_commits",
+ "sourceSchemaName": "Default",
+ "lineWidth": 1,
+ "visible": true,
+ "vertices": [
+ {
+ "x": 883,
+ "y": 1473
+ },
+ {
+ "x": 966,
+ "y": 1473
+ }
+ ],
+ "label": {
+ "x": 873,
+ "y": 1434,
+ "width": 202,
+ "height": 31,
+ "fontName": "Arial Unicode MS",
+ "fontSize": 14,
+ "fontColor": {
+ "r": 51,
+ "g": 51,
+ "b": 51,
+ "a": 1
+ },
+ "isFontBold": false,
+ "isFontItalic": false,
+ "isVisible": false
+ }
+ },
+ {
+ "name": "fk_ref_commits_ref_commits_2",
+ "sourceTableName": "ref_commits",
+ "sourceSchemaName": "Default",
+ "lineWidth": 1,
+ "visible": true,
+ "vertices": [
+ {
+ "x": 883,
+ "y": 1450
+ },
+ {
+ "x": 966,
+ "y": 1448
+ }
+ ],
+ "label": {
+ "x": 873,
+ "y": 1411,
+ "width": 202,
+ "height": 31,
+ "fontName": "Arial Unicode MS",
+ "fontSize": 14,
+ "fontColor": {
+ "r": 51,
+ "g": 51,
+ "b": 51,
+ "a": 1
+ },
+ "isFontBold": false,
+ "isFontItalic": false,
+ "isVisible": false
+ }
}
],
"viewRelations": []
diff --git a/static/img/DomainLayerSchema/schema-diagram.png b/static/img/DomainLayerSchema/schema-diagram.png
index b9bef6a81..af04b66e5 100644
Binary files a/static/img/DomainLayerSchema/schema-diagram.png and b/static/img/DomainLayerSchema/schema-diagram.png differ
[incubator-devlake-website] 01/03: docs: update refdiff docs
Posted by zk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
commit d1075c24bd0b64534dd46fead83377d9a59fad40
Author: abeizn <zi...@merico.dev>
AuthorDate: Wed Nov 23 15:07:29 2022 +0800
docs: update refdiff docs
---
docs/Plugins/refdiff.md | 79 +++++++++++++++++++++++++++----------------------
1 file changed, 44 insertions(+), 35 deletions(-)
diff --git a/docs/Plugins/refdiff.md b/docs/Plugins/refdiff.md
index d97f1c1da..4ba983c16 100644
--- a/docs/Plugins/refdiff.md
+++ b/docs/Plugins/refdiff.md
@@ -4,10 +4,9 @@ description: >
RefDiff Plugin
---
-
## Summary
-For development workload analysis, we often need to know how many commits have been created between 2 releases. This plugin calculates which commits differ between 2 Ref (branch/tag), and the result will be stored back into the database for further analysis.
+For development workload analysis, we often need to know how many commits have been created between 2 releases. This plugin calculates which commits differ between 2 commits, and the commits are produced by Ref (branch/tag) or by deployments, and the result will be stored back into the database for further analysis.
## Important Note
@@ -22,22 +21,25 @@ This is an enrichment plugin based on the domain layer data, no configuration is
To trigger the enrichment, you need to insert a new task into your pipeline.
1. Make sure `commits` and `refs` are collected into your database, `refs` table should contain records like following:
- ```
- id ref_type
- github:GithubRepo:384111310:refs/tags/0.3.5 TAG
- github:GithubRepo:384111310:refs/tags/0.3.6 TAG
- github:GithubRepo:384111310:refs/tags/0.5.0 TAG
- github:GithubRepo:384111310:refs/tags/v0.0.1 TAG
- github:GithubRepo:384111310:refs/tags/v0.2.0 TAG
- github:GithubRepo:384111310:refs/tags/v0.3.0 TAG
- github:GithubRepo:384111310:refs/tags/v0.4.0 TAG
- github:GithubRepo:384111310:refs/tags/v0.6.0 TAG
- github:GithubRepo:384111310:refs/tags/v0.6.1 TAG
- ```
+ ```
+ id ref_type
+ github:GithubRepo:1:384111310:refs/tags/0.3.5 TAG
+ github:GithubRepo:1:384111310:refs/tags/0.3.6 TAG
+ github:GithubRepo:1:384111310:refs/tags/0.5.0 TAG
+ github:GithubRepo:1:384111310:refs/tags/v0.0.1 TAG
+ github:GithubRepo:1:384111310:refs/tags/v0.2.0 TAG
+ github:GithubRepo:1:384111310:refs/tags/v0.3.0 TAG
+ github:GithubRepo:1:384111310:refs/tags/v0.4.0 TAG
+ github:GithubRepo:1:384111310:refs/tags/v0.6.0 TAG
+ github:GithubRepo:1:384111310:refs/tags/v0.6.1 TAG
+ ```
2. If you want to run calculatePrCherryPick, please configure GITHUB_PR_TITLE_PATTERN in .env, you can check the example in .env.example(we have a default value, please make sure your pattern is disclosed by single quotes '')
3. And then, trigger a pipeline like the following format, you can also define sub-tasks, calculateRefDiff will calculate commits between two refs, and creatRefBugStats will create a table to show a bug list between two refs:
+
```shell
-curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
+curl 'http://localhost:8080/pipelines' \
+--header 'Content-Type: application/json' \
+--data-raw '
{
"name": "test-refdiff",
"tasks": [
@@ -45,7 +47,7 @@ curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
{
"plugin": "refdiff",
"options": {
- "repoId": "github:GithubRepo:384111310",
+ "repoId": "github:GithubRepo:1:384111310",
"pairs": [
{ "newRef": "refs/tags/v0.6.0", "oldRef": "refs/tags/0.5.0" },
{ "newRef": "refs/tags/0.5.0", "oldRef": "refs/tags/0.4.0" }
@@ -59,12 +61,15 @@ curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
}
]
]
-}
-JSON
+}'
```
+
Or if you preferred calculating latest releases
+
```shell
-curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
+curl 'http://localhost:8080/pipelines' \
+--header 'Content-Type: application/json' \
+--data-raw '
{
"name": "test-refdiff",
"tasks": [
@@ -72,7 +77,7 @@ curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
{
"plugin": "refdiff",
"options": {
- "repoId": "github:GithubRepo:384111310",
+ "repoId": "github:GithubRepo:1:384111310",
"tagsPattern": "v\d+\.\d+.\d+",
"tagsLimit": 10,
"tagsOrder": "reverse semver",
@@ -85,15 +90,17 @@ curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
}
]
]
-}
-JSON
+}'
```
## How to use refdiff in DORA
+
RefDiff can be called by the [DORA plugin](https://github.com/apache/incubator-devlake/tree/main/plugins/dora) to support the calculation of [DORA metrics](https://devlake.apache.org/docs/UserManuals/DORA). RefDiff has a subtask called 'calculateProjectDeploymentCommitsDiff'. This subtask takes the `project_name` from task options to calculate the commits diff between two consecutive deployments in this project. That is to say, refdiff will generate the relationship between `deployed com [...]
-
+
```shell
-curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
+curl 'http://localhost:8080/pipelines' \
+--header 'Content-Type: application/json' \
+--data-raw '
{
"name": "test-refdiff-dora",
"tasks": [
@@ -101,7 +108,7 @@ curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
{
"plugin": "refdiff",
"options": {
- "projectName": "xxxx",
+ "projectName": "project_name_1",
"tasks": [
"calculateProjectDeploymentCommitsDiff"
]
@@ -109,7 +116,7 @@ curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
}
]
]
-}
+}'
```
## Development
@@ -126,7 +133,7 @@ apt install cmake -y
[centos]
yum install cmake -y
-2. compiling
+2. compiling
git clone -b v1.3.0 https://github.com/libgit2/libgit2.git && cd libgit2
mkdir build && cd build && cmake ..
make && make install
@@ -142,25 +149,26 @@ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
#### Troubleshooting (Linux)
-> Q: # pkg-config --cflags -- libgit2 Package libgit2 was not found in the pkg-config search path.
-Perhaps you should add the directory containing `libgit2.pc` to the PKG_CONFIG_PATH environment variable
-No package 'libgit2' found pkg-config: exit status 1
+> Q: # pkg-config --cflags -- libgit2 Package libgit2 was not found in the pkg-config search path.
+> Perhaps you should add the directory containing `libgit2.pc` to the PKG_CONFIG_PATH environment variable
+> No package 'libgit2' found pkg-config: exit status 1
> A:
> Make sure your pkg config path covers the installation:
> if your libgit2.pc in `/usr/local/lib64/pkgconfig`(like centos)
->
+>
> `export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib64:/usr/local/lib64/pkgconfig`
->
+>
> else if your libgit2.pc in `/usr/local/lib/pkgconfig`(like ubuntu)
->
+>
> `export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig`
->
+>
> else consider install pkgconfig or rebuild the libgit2
### MacOS
NOTE: **Do NOT** install libgit2 via `MacPorts` or `homebrew`, install from source instead.
+
```
brew install cmake
git clone https://github.com/libgit2/libgit2.git
@@ -178,11 +186,12 @@ make install
> Q: I got an error saying: `pkg-config: exec: "pkg-config": executable file not found in $PATH`
> A:
+>
> 1. Make sure you have pkg-config installed:
>
> `brew install pkg-config`
>
> 2. Make sure your pkg config path covers the installation:
-> `export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig`
+> `export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig`
<br/><br/><br/>