You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by GitBox <gi...@apache.org> on 2022/12/26 07:18:32 UTC
[GitHub] [incubator-devlake-website] klesh commented on a diff in pull request #308: Project framework
klesh commented on code in PR #308:
URL: https://github.com/apache/incubator-devlake-website/pull/308#discussion_r1057099894
##########
docs/DeveloperManuals/Project.md:
##########
@@ -0,0 +1,184 @@
+---
+title: "Project"
+sidebar_position: 5
+description: >
+ `Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculations like `Dora`.
+---
+
+# Summary
+
+For some metric calculations such as the `DORA` metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.
+
+For example, we may use `Gitlab` for code version control, `Jenkins` for CI/CD, to calculate PR deployment cycle time, we need to know which `Gitlab Projects` and `Jenkins Jobs` are related for correctness and performance reasons.
+
+However, in most cases, we have multiple `Gitlab Projects` / `Jenkins Jobs` that belong to different teams/agendas in our Apache DevLake database.
+
+To distinguish them into different groups. The `Project` is introduced in v0.15. Essentially, a `project` consists of a set of [Scopes](../Overview/KeyConcepts.md#data-scope), i.e., a couple of `Gitlab Projects`, `Jira Boards` or `Jenkins Jobs`, etc.
+
+`Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculation like `Dora`.
+
+Next, let us introduce `Project` in the following order:
+- `Project` related models
+- Related APIs that can be used to manipulate `Project` models
+- The interface that needs to be implemented when developing various plugins to support the `Project`.
+ - The interface that needs to be implemented to develop the `Data Source Plugin`
+ - The interface that needs to be implemented to develop the `Metrics Plugin`
+
+# Models
+
+To support project we contains the following three models:
+ - `projects` describes a project object, including its name, creation and update time and other basic information
+ - `project_metric_settings` describes what metrics plugin a project had enabled.
+ - `project_mapping` describes the mapping relationship of project and scope, including the name of the project、table and row_id.
+
+## projects
+
+| **field** | **type** | **length** | **description** | **key** |
+| ------------- | -------- | ---------- | ----------------------------- | ------- |
+| `name` | varchar | 255 | name for project | PK |
+| `description` | longtext | | description of the project | |
+| `created_at` | datetime | 3 | created time of project | |
+| `updated_at` | datetime | 3 | last updated time of project | |
+
+
+| **name** | **describe** | **created_at** | **updated_at** |
+| --------- | ------------------------------------ | ----------------------- | ------------------------|
+| project_1 | this is one of the test projects | 2022-11-01 01:22:13.000 | 2022-11-01 02:24:15.000 |
+| project_2 | this is another project test project | 2022-11-01 01:23:29.000 | 2022-11-01 02:27:24.000 |
+
+## project_metric_settings
+
+| **field** | **type** | **length** | **description** | **key** |
+| --------------- | -------- | ---------- | ---------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `plugin_name` | varchar | 255 | name for plugin | PK |
+| `plugin_option` | longtext | | check if metric plugins have been enabled by the project | |
+| `enable` | tinyint | 1 | if the metric plugins is enabled | |
+
+
+| **project_name** | **plugin_name** | **plugin_option** | **enable** |
+| ---------------- | --------------- | ----------------- | ---------- |
+| project_1 | dora | {} | true |
+| project_2 | dora | {} | false |
+
+## project_mapping
+
+| **field** | **type** | **length** | **description** | **key** |
+| -------------- | -------- | ---------- | ------------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `table` | varchar | 255 | the table name of [Scope](../Overview/KeyConcepts.md#data-scope) | PK |
+| `row_id` | varchar | 255 | the row_id in the [Scope](../Overview/KeyConcepts.md#data-scope) table | PK |
+
+
+| **project_name** | **table** | **row_id** |
+| ---------------- | --------- | ------------------------ |
+| project_1 | Repo | gitlab:GithubRepo:1:lake |
+| project_1 | Board | jira:JiraBoard:1:lake |
+| project_2 | Repo | github:GithubRepo:1:lake |
+
+## _devlake_blueprint
+
+By adding the `project_name` field to the `blueprint` Models, we associate `project` with `blueprint` in one-to-one manner.
+# How to manage project via API
+
+For API specification, please check the swagger doc(by visiting `[Your Config-UI Host]/api/swagger/index.html`).
+Related endpoints:
+
+1. /projects
+2. /projects/:projectName/metrics
+3. /plugins
+
+# The interface that needs to be implemented
+
+As a developer, when we are doing a new plugin development, or when we do some other operations that need to be associated with `Project` to add, delete, and modify operations, we need to have a detailed understanding of `Project`.
Review Comment:
I feel this paragraph is not necessary. Viewers would determine whether they should learn it or not.
##########
docs/DeveloperManuals/Project.md:
##########
@@ -0,0 +1,184 @@
+---
+title: "Project"
+sidebar_position: 5
+description: >
+ `Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculations like `Dora`.
+---
+
+# Summary
+
+For some metric calculations such as the `DORA` metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.
+
+For example, we may use `Gitlab` for code version control, `Jenkins` for CI/CD, to calculate PR deployment cycle time, we need to know which `Gitlab Projects` and `Jenkins Jobs` are related for correctness and performance reasons.
+
+However, in most cases, we have multiple `Gitlab Projects` / `Jenkins Jobs` that belong to different teams/agendas in our Apache DevLake database.
+
+To distinguish them into different groups. The `Project` is introduced in v0.15. Essentially, a `project` consists of a set of [Scopes](../Overview/KeyConcepts.md#data-scope), i.e., a couple of `Gitlab Projects`, `Jira Boards` or `Jenkins Jobs`, etc.
+
+`Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculation like `Dora`.
+
+Next, let us introduce `Project` in the following order:
+- `Project` related models
+- Related APIs that can be used to manipulate `Project` models
+- The interface that needs to be implemented when developing various plugins to support the `Project`.
+ - The interface that needs to be implemented to develop the `Data Source Plugin`
+ - The interface that needs to be implemented to develop the `Metrics Plugin`
+
+# Models
+
+To support project we contains the following three models:
+ - `projects` describes a project object, including its name, creation and update time and other basic information
+ - `project_metric_settings` describes what metrics plugin a project had enabled.
+ - `project_mapping` describes the mapping relationship of project and scope, including the name of the project、table and row_id.
+
+## projects
+
+| **field** | **type** | **length** | **description** | **key** |
+| ------------- | -------- | ---------- | ----------------------------- | ------- |
+| `name` | varchar | 255 | name for project | PK |
+| `description` | longtext | | description of the project | |
+| `created_at` | datetime | 3 | created time of project | |
+| `updated_at` | datetime | 3 | last updated time of project | |
+
+
+| **name** | **describe** | **created_at** | **updated_at** |
+| --------- | ------------------------------------ | ----------------------- | ------------------------|
+| project_1 | this is one of the test projects | 2022-11-01 01:22:13.000 | 2022-11-01 02:24:15.000 |
+| project_2 | this is another project test project | 2022-11-01 01:23:29.000 | 2022-11-01 02:27:24.000 |
+
+## project_metric_settings
+
+| **field** | **type** | **length** | **description** | **key** |
+| --------------- | -------- | ---------- | ---------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `plugin_name` | varchar | 255 | name for plugin | PK |
+| `plugin_option` | longtext | | check if metric plugins have been enabled by the project | |
+| `enable` | tinyint | 1 | if the metric plugins is enabled | |
+
+
+| **project_name** | **plugin_name** | **plugin_option** | **enable** |
+| ---------------- | --------------- | ----------------- | ---------- |
+| project_1 | dora | {} | true |
+| project_2 | dora | {} | false |
+
+## project_mapping
+
+| **field** | **type** | **length** | **description** | **key** |
+| -------------- | -------- | ---------- | ------------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `table` | varchar | 255 | the table name of [Scope](../Overview/KeyConcepts.md#data-scope) | PK |
+| `row_id` | varchar | 255 | the row_id in the [Scope](../Overview/KeyConcepts.md#data-scope) table | PK |
+
+
+| **project_name** | **table** | **row_id** |
+| ---------------- | --------- | ------------------------ |
+| project_1 | Repo | gitlab:GithubRepo:1:lake |
+| project_1 | Board | jira:JiraBoard:1:lake |
+| project_2 | Repo | github:GithubRepo:1:lake |
+
+## _devlake_blueprint
+
+By adding the `project_name` field to the `blueprint` Models, we associate `project` with `blueprint` in one-to-one manner.
+# How to manage project via API
+
+For API specification, please check the swagger doc(by visiting `[Your Config-UI Host]/api/swagger/index.html`).
+Related endpoints:
+
+1. /projects
+2. /projects/:projectName/metrics
+3. /plugins
+
+# The interface that needs to be implemented
+
+As a developer, when we are doing a new plugin development, or when we do some other operations that need to be associated with `Project` to add, delete, and modify operations, we need to have a detailed understanding of `Project`.
+
+We divide plugins into three categories
+- The second category is `Data Source Plugin`, such as `GitLab` `GitHub` `Jira` `Jenkins`, etc. These plugins collect data from various data sources
+- The third category is `Metrics Plugin`, such as `Dora`, etc. These plug-ins do not directly contact the data source but do secondary calculations based on
+
+the collected data after the `Data Source Plugin` works
Review Comment:
what?
##########
docs/DeveloperManuals/Project.md:
##########
@@ -0,0 +1,184 @@
+---
+title: "Project"
+sidebar_position: 5
+description: >
+ `Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculations like `Dora`.
+---
+
+# Summary
+
+For some metric calculations such as the `DORA` metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.
+
+For example, we may use `Gitlab` for code version control, `Jenkins` for CI/CD, to calculate PR deployment cycle time, we need to know which `Gitlab Projects` and `Jenkins Jobs` are related for correctness and performance reasons.
+
+However, in most cases, we have multiple `Gitlab Projects` / `Jenkins Jobs` that belong to different teams/agendas in our Apache DevLake database.
+
+To distinguish them into different groups. The `Project` is introduced in v0.15. Essentially, a `project` consists of a set of [Scopes](../Overview/KeyConcepts.md#data-scope), i.e., a couple of `Gitlab Projects`, `Jira Boards` or `Jenkins Jobs`, etc.
+
+`Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculation like `Dora`.
+
+Next, let us introduce `Project` in the following order:
+- `Project` related models
+- Related APIs that can be used to manipulate `Project` models
+- The interface that needs to be implemented when developing various plugins to support the `Project`.
+ - The interface that needs to be implemented to develop the `Data Source Plugin`
+ - The interface that needs to be implemented to develop the `Metrics Plugin`
+
+# Models
+
+To support project we contains the following three models:
+ - `projects` describes a project object, including its name, creation and update time and other basic information
+ - `project_metric_settings` describes what metrics plugin a project had enabled.
+ - `project_mapping` describes the mapping relationship of project and scope, including the name of the project、table and row_id.
+
+## projects
+
+| **field** | **type** | **length** | **description** | **key** |
+| ------------- | -------- | ---------- | ----------------------------- | ------- |
+| `name` | varchar | 255 | name for project | PK |
+| `description` | longtext | | description of the project | |
+| `created_at` | datetime | 3 | created time of project | |
+| `updated_at` | datetime | 3 | last updated time of project | |
+
+
+| **name** | **describe** | **created_at** | **updated_at** |
+| --------- | ------------------------------------ | ----------------------- | ------------------------|
+| project_1 | this is one of the test projects | 2022-11-01 01:22:13.000 | 2022-11-01 02:24:15.000 |
+| project_2 | this is another project test project | 2022-11-01 01:23:29.000 | 2022-11-01 02:27:24.000 |
+
+## project_metric_settings
+
+| **field** | **type** | **length** | **description** | **key** |
+| --------------- | -------- | ---------- | ---------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `plugin_name` | varchar | 255 | name for plugin | PK |
+| `plugin_option` | longtext | | check if metric plugins have been enabled by the project | |
+| `enable` | tinyint | 1 | if the metric plugins is enabled | |
+
+
+| **project_name** | **plugin_name** | **plugin_option** | **enable** |
+| ---------------- | --------------- | ----------------- | ---------- |
+| project_1 | dora | {} | true |
+| project_2 | dora | {} | false |
+
+## project_mapping
+
+| **field** | **type** | **length** | **description** | **key** |
+| -------------- | -------- | ---------- | ------------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `table` | varchar | 255 | the table name of [Scope](../Overview/KeyConcepts.md#data-scope) | PK |
+| `row_id` | varchar | 255 | the row_id in the [Scope](../Overview/KeyConcepts.md#data-scope) table | PK |
+
+
+| **project_name** | **table** | **row_id** |
+| ---------------- | --------- | ------------------------ |
+| project_1 | Repo | gitlab:GithubRepo:1:lake |
+| project_1 | Board | jira:JiraBoard:1:lake |
+| project_2 | Repo | github:GithubRepo:1:lake |
+
+## _devlake_blueprint
+
+By adding the `project_name` field to the `blueprint` Models, we associate `project` with `blueprint` in one-to-one manner.
+# How to manage project via API
+
+For API specification, please check the swagger doc(by visiting `[Your Config-UI Host]/api/swagger/index.html`).
+Related endpoints:
+
+1. /projects
+2. /projects/:projectName/metrics
+3. /plugins
+
+# The interface that needs to be implemented
+
+As a developer, when we are doing a new plugin development, or when we do some other operations that need to be associated with `Project` to add, delete, and modify operations, we need to have a detailed understanding of `Project`.
+
+We divide plugins into three categories
+- The second category is `Data Source Plugin`, such as `GitLab` `GitHub` `Jira` `Jenkins`, etc. These plugins collect data from various data sources
+- The third category is `Metrics Plugin`, such as `Dora`, etc. These plug-ins do not directly contact the data source but do secondary calculations based on
+
+the collected data after the `Data Source Plugin` works
+
+The second and third types of plug-ins here need to implement the corresponding interface to support the project.
+
+## Data Source Plugin
+
+`Data Source Plugin` is a plugin used to extract data from a data source.
+
+For example `GitLab` `GitHub` `Jira` `Jenkins` etc.
+
+These plugins, from various data sources, extract data into the database and store them, they deal directly with the data source, so we classify them as `Data Source Plugin`.
+
+## the DataSourcePluginBlueprintV200 interface
+
+`Data Source Plugin` needs to implement `DataSourcePluginBlueprintV200` interface to support `project`
+
+The interface definition for this interface is as follows
+
+```go
+// DataSourcePluginBlueprintV200 extends the V100 to provide support for
+// Project, so that complex metrics like DORA can be implemented based on a set
+// of Data Scopes
+type DataSourcePluginBlueprintV200 interface {
+ MakeDataSourcePipelinePlanV200(
+ connectionId uint64,
+ scopes []*BlueprintScopeV200,
+ syncPolicy BlueprintSyncPolicy,
+ ) (PipelinePlan, []Scope, errors.Error)
+}
+```
+
+`project` needs to provide a specific set of [Scopes](../Overview/KeyConcepts.md#data-scope) for a specific `connection` to the plug-in through this interface, and then obtain the plug-in involved in the `PipelineTask` All `plugins` and corresponding parameter information. At the same time, the plug-in needs to convert entities like `repo` and `board` in the data source into a `scope interface` that `project` can understand
Review Comment:
The `DataSourcePluginBlueprintV200` is introduced for the plugin to tell the framework how should the pipeline to be executed and what specific `Domain Layer Scopes` would be produced based on users input (`connectionId`, `Tool Layer Scopes` and `SyncPolicy`)
##########
docs/DeveloperManuals/Project.md:
##########
@@ -0,0 +1,184 @@
+---
+title: "Project"
+sidebar_position: 5
+description: >
+ `Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculations like `Dora`.
+---
+
+# Summary
+
+For some metric calculations such as the `DORA` metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.
+
+For example, we may use `Gitlab` for code version control, `Jenkins` for CI/CD, to calculate PR deployment cycle time, we need to know which `Gitlab Projects` and `Jenkins Jobs` are related for correctness and performance reasons.
+
+However, in most cases, we have multiple `Gitlab Projects` / `Jenkins Jobs` that belong to different teams/agendas in our Apache DevLake database.
+
+To distinguish them into different groups. The `Project` is introduced in v0.15. Essentially, a `project` consists of a set of [Scopes](../Overview/KeyConcepts.md#data-scope), i.e., a couple of `Gitlab Projects`, `Jira Boards` or `Jenkins Jobs`, etc.
+
+`Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculation like `Dora`.
+
+Next, let us introduce `Project` in the following order:
+- `Project` related models
+- Related APIs that can be used to manipulate `Project` models
+- The interface that needs to be implemented when developing various plugins to support the `Project`.
+ - The interface that needs to be implemented to develop the `Data Source Plugin`
+ - The interface that needs to be implemented to develop the `Metrics Plugin`
+
+# Models
+
+To support project we contains the following three models:
+ - `projects` describes a project object, including its name, creation and update time and other basic information
+ - `project_metric_settings` describes what metrics plugin a project had enabled.
+ - `project_mapping` describes the mapping relationship of project and scope, including the name of the project、table and row_id.
+
+## projects
+
+| **field** | **type** | **length** | **description** | **key** |
+| ------------- | -------- | ---------- | ----------------------------- | ------- |
+| `name` | varchar | 255 | name for project | PK |
+| `description` | longtext | | description of the project | |
+| `created_at` | datetime | 3 | created time of project | |
+| `updated_at` | datetime | 3 | last updated time of project | |
+
+
+| **name** | **describe** | **created_at** | **updated_at** |
+| --------- | ------------------------------------ | ----------------------- | ------------------------|
+| project_1 | this is one of the test projects | 2022-11-01 01:22:13.000 | 2022-11-01 02:24:15.000 |
+| project_2 | this is another project test project | 2022-11-01 01:23:29.000 | 2022-11-01 02:27:24.000 |
+
+## project_metric_settings
+
+| **field** | **type** | **length** | **description** | **key** |
+| --------------- | -------- | ---------- | ---------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `plugin_name` | varchar | 255 | name for plugin | PK |
+| `plugin_option` | longtext | | check if metric plugins have been enabled by the project | |
+| `enable` | tinyint | 1 | if the metric plugins is enabled | |
+
+
+| **project_name** | **plugin_name** | **plugin_option** | **enable** |
+| ---------------- | --------------- | ----------------- | ---------- |
+| project_1 | dora | {} | true |
+| project_2 | dora | {} | false |
+
+## project_mapping
+
+| **field** | **type** | **length** | **description** | **key** |
+| -------------- | -------- | ---------- | ------------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `table` | varchar | 255 | the table name of [Scope](../Overview/KeyConcepts.md#data-scope) | PK |
+| `row_id` | varchar | 255 | the row_id in the [Scope](../Overview/KeyConcepts.md#data-scope) table | PK |
+
+
+| **project_name** | **table** | **row_id** |
+| ---------------- | --------- | ------------------------ |
+| project_1 | Repo | gitlab:GithubRepo:1:lake |
+| project_1 | Board | jira:JiraBoard:1:lake |
+| project_2 | Repo | github:GithubRepo:1:lake |
+
+## _devlake_blueprint
+
+By adding the `project_name` field to the `blueprint` Models, we associate `project` with `blueprint` in one-to-one manner.
+# How to manage project via API
+
+For API specification, please check the swagger doc(by visiting `[Your Config-UI Host]/api/swagger/index.html`).
+Related endpoints:
+
+1. /projects
+2. /projects/:projectName/metrics
+3. /plugins
+
+# The interface that needs to be implemented
+
+As a developer, when we are doing a new plugin development, or when we do some other operations that need to be associated with `Project` to add, delete, and modify operations, we need to have a detailed understanding of `Project`.
+
+We divide plugins into three categories
+- The second category is `Data Source Plugin`, such as `GitLab` `GitHub` `Jira` `Jenkins`, etc. These plugins collect data from various data sources
+- The third category is `Metrics Plugin`, such as `Dora`, etc. These plug-ins do not directly contact the data source but do secondary calculations based on
+
+the collected data after the `Data Source Plugin` works
+
+The second and third types of plug-ins here need to implement the corresponding interface to support the project.
+
+## Data Source Plugin
+
+`Data Source Plugin` is a plugin used to extract data from a data source.
+
+For example `GitLab` `GitHub` `Jira` `Jenkins` etc.
+
+These plugins, from various data sources, extract data into the database and store them, they deal directly with the data source, so we classify them as `Data Source Plugin`.
+
+## the DataSourcePluginBlueprintV200 interface
+
+`Data Source Plugin` needs to implement `DataSourcePluginBlueprintV200` interface to support `project`
+
+The interface definition for this interface is as follows
+
+```go
+// DataSourcePluginBlueprintV200 extends the V100 to provide support for
+// Project, so that complex metrics like DORA can be implemented based on a set
+// of Data Scopes
+type DataSourcePluginBlueprintV200 interface {
+ MakeDataSourcePipelinePlanV200(
+ connectionId uint64,
+ scopes []*BlueprintScopeV200,
+ syncPolicy BlueprintSyncPolicy,
+ ) (PipelinePlan, []Scope, errors.Error)
+}
+```
+
+`project` needs to provide a specific set of [Scopes](../Overview/KeyConcepts.md#data-scope) for a specific `connection` to the plug-in through this interface, and then obtain the plug-in involved in the `PipelineTask` All `plugins` and corresponding parameter information. At the same time, the plug-in needs to convert entities like `repo` and `board` in the data source into a `scope interface` that `project` can understand
+
+The corresponding `scope interface` has been implemented at following files of in the framework layer:
+- `models/domainlayer/ticket/board.go`
+- `models/domainlayer/code/repo.go`
+
+In the `plugins/gitlab/impl/impl.go` file, there is a `Gitlab` plugin implementation of the above interface, which can be used as a sample reference.
+
+And the `plugins/gitlab/api/blueprint_v200.go` contains implementation details.
+
+The following files contain the models that the relevant implementations depend on for reference:
+- `plugins/gitlab/models/project.go`
+- `plugins/gitlab/models/transformation_rule.go`
+
+## Metrics Plugin
+The `Metrics Plugin` is based on the data collected by the `Data Source Plugin` to do secondary calculations.
+
+For example `Dora`, and `Refdff` plugins belong to the `Metrics Plugin`
Review Comment:
already mentioned above, do not repeat.
##########
docs/DeveloperManuals/Project.md:
##########
@@ -0,0 +1,184 @@
+---
+title: "Project"
+sidebar_position: 5
+description: >
+ `Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculations like `Dora`.
+---
+
+# Summary
+
+For some metric calculations such as the `DORA` metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.
+
+For example, we may use `Gitlab` for code version control, `Jenkins` for CI/CD, to calculate PR deployment cycle time, we need to know which `Gitlab Projects` and `Jenkins Jobs` are related for correctness and performance reasons.
+
+However, in most cases, we have multiple `Gitlab Projects` / `Jenkins Jobs` that belong to different teams/agendas in our Apache DevLake database.
+
+To distinguish them into different groups. The `Project` is introduced in v0.15. Essentially, a `project` consists of a set of [Scopes](../Overview/KeyConcepts.md#data-scope), i.e., a couple of `Gitlab Projects`, `Jira Boards` or `Jenkins Jobs`, etc.
+
+`Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculation like `Dora`.
+
+Next, let us introduce `Project` in the following order:
+- `Project` related models
+- Related APIs that can be used to manipulate `Project` models
+- The interface that needs to be implemented when developing various plugins to support the `Project`.
+ - The interface that needs to be implemented to develop the `Data Source Plugin`
+ - The interface that needs to be implemented to develop the `Metrics Plugin`
+
+# Models
+
+To support project we contains the following three models:
+ - `projects` describes a project object, including its name, creation and update time and other basic information
+ - `project_metric_settings` describes what metrics plugin a project had enabled.
+ - `project_mapping` describes the mapping relationship of project and scope, including the name of the project、table and row_id.
+
+## projects
+
+| **field** | **type** | **length** | **description** | **key** |
+| ------------- | -------- | ---------- | ----------------------------- | ------- |
+| `name` | varchar | 255 | name for project | PK |
+| `description` | longtext | | description of the project | |
+| `created_at` | datetime | 3 | created time of project | |
+| `updated_at` | datetime | 3 | last updated time of project | |
+
+
+| **name** | **describe** | **created_at** | **updated_at** |
+| --------- | ------------------------------------ | ----------------------- | ------------------------|
+| project_1 | this is one of the test projects | 2022-11-01 01:22:13.000 | 2022-11-01 02:24:15.000 |
+| project_2 | this is another project test project | 2022-11-01 01:23:29.000 | 2022-11-01 02:27:24.000 |
+
+## project_metric_settings
+
+| **field** | **type** | **length** | **description** | **key** |
+| --------------- | -------- | ---------- | ---------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `plugin_name` | varchar | 255 | name for plugin | PK |
+| `plugin_option` | longtext | | check if metric plugins have been enabled by the project | |
+| `enable` | tinyint | 1 | if the metric plugins is enabled | |
+
+
+| **project_name** | **plugin_name** | **plugin_option** | **enable** |
+| ---------------- | --------------- | ----------------- | ---------- |
+| project_1 | dora | {} | true |
+| project_2 | dora | {} | false |
+
+## project_mapping
+
+| **field** | **type** | **length** | **description** | **key** |
+| -------------- | -------- | ---------- | ------------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `table` | varchar | 255 | the table name of [Scope](../Overview/KeyConcepts.md#data-scope) | PK |
+| `row_id` | varchar | 255 | the row_id in the [Scope](../Overview/KeyConcepts.md#data-scope) table | PK |
+
+
+| **project_name** | **table** | **row_id** |
+| ---------------- | --------- | ------------------------ |
+| project_1 | Repo | gitlab:GithubRepo:1:lake |
+| project_1 | Board | jira:JiraBoard:1:lake |
+| project_2 | Repo | github:GithubRepo:1:lake |
+
+## _devlake_blueprint
+
+By adding the `project_name` field to the `blueprint` Models, we associate `project` with `blueprint` in one-to-one manner.
+# How to manage project via API
+
+For API specification, please check the swagger doc(by visiting `[Your Config-UI Host]/api/swagger/index.html`).
+Related endpoints:
+
+1. /projects
+2. /projects/:projectName/metrics
+3. /plugins
+
+# The interface that needs to be implemented
+
+As a developer, when we are doing a new plugin development, or when we do some other operations that need to be associated with `Project` to add, delete, and modify operations, we need to have a detailed understanding of `Project`.
+
+We divide plugins into three categories
+- The second category is `Data Source Plugin`, such as `GitLab` `GitHub` `Jira` `Jenkins`, etc. These plugins collect data from various data sources
+- The third category is `Metrics Plugin`, such as `Dora`, etc. These plug-ins do not directly contact the data source but do secondary calculations based on
+
+the collected data after the `Data Source Plugin` works
+
+The second and third types of plug-ins here need to implement the corresponding interface to support the project.
+
+## Data Source Plugin
+
+`Data Source Plugin` is a plugin used to extract data from a data source.
+
+For example `GitLab` `GitHub` `Jira` `Jenkins` etc.
+
+These plugins, from various data sources, extract data into the database and store them, they deal directly with the data source, so we classify them as `Data Source Plugin`.
+
+## the DataSourcePluginBlueprintV200 interface
+
+`Data Source Plugin` needs to implement `DataSourcePluginBlueprintV200` interface to support `project`
+
+The interface definition for this interface is as follows
+
+```go
+// DataSourcePluginBlueprintV200 extends the V100 to provide support for
+// Project, so that complex metrics like DORA can be implemented based on a set
+// of Data Scopes
+type DataSourcePluginBlueprintV200 interface {
+ MakeDataSourcePipelinePlanV200(
+ connectionId uint64,
+ scopes []*BlueprintScopeV200,
+ syncPolicy BlueprintSyncPolicy,
+ ) (PipelinePlan, []Scope, errors.Error)
+}
+```
+
+`project` needs to provide a specific set of [Scopes](../Overview/KeyConcepts.md#data-scope) for a specific `connection` to the plug-in through this interface, and then obtain the plug-in involved in the `PipelineTask` All `plugins` and corresponding parameter information. At the same time, the plug-in needs to convert entities like `repo` and `board` in the data source into a `scope interface` that `project` can understand
+
+The corresponding `scope interface` has been implemented at following files of in the framework layer:
+- `models/domainlayer/ticket/board.go`
+- `models/domainlayer/code/repo.go`
+
+In the `plugins/gitlab/impl/impl.go` file, there is a `Gitlab` plugin implementation of the above interface, which can be used as a sample reference.
+
+And the `plugins/gitlab/api/blueprint_v200.go` contains implementation details.
+
+The following files contain the models that the relevant implementations depend on for reference:
+- `plugins/gitlab/models/project.go`
+- `plugins/gitlab/models/transformation_rule.go`
+
+## Metrics Plugin
+The `Metrics Plugin` is based on the data collected by the `Data Source Plugin` to do secondary calculations.
Review Comment:
already mentioned above, do not repeat.
##########
docs/DeveloperManuals/Project.md:
##########
@@ -0,0 +1,184 @@
+---
+title: "Project"
+sidebar_position: 5
+description: >
+ `Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculations like `Dora`.
+---
+
+# Summary
+
+For some metric calculations such as the `DORA` metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.
+
+For example, we may use `Gitlab` for code version control, `Jenkins` for CI/CD, to calculate PR deployment cycle time, we need to know which `Gitlab Projects` and `Jenkins Jobs` are related for correctness and performance reasons.
+
+However, in most cases, we have multiple `Gitlab Projects` / `Jenkins Jobs` that belong to different teams/agendas in our Apache DevLake database.
+
+To distinguish them into different groups. The `Project` is introduced in v0.15. Essentially, a `project` consists of a set of [Scopes](../Overview/KeyConcepts.md#data-scope), i.e., a couple of `Gitlab Projects`, `Jira Boards` or `Jenkins Jobs`, etc.
+
+`Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculation like `Dora`.
+
+Next, let us introduce `Project` in the following order:
+- `Project` related models
+- Related APIs that can be used to manipulate `Project` models
+- The interface that needs to be implemented when developing various plugins to support the `Project`.
+ - The interface that needs to be implemented to develop the `Data Source Plugin`
+ - The interface that needs to be implemented to develop the `Metrics Plugin`
+
+# Models
+
+To support project we contains the following three models:
+ - `projects` describes a project object, including its name, creation and update time and other basic information
+ - `project_metric_settings` describes what metrics plugin a project had enabled.
+ - `project_mapping` describes the mapping relationship of project and scope, including the name of the project、table and row_id.
+
+## projects
+
+| **field** | **type** | **length** | **description** | **key** |
+| ------------- | -------- | ---------- | ----------------------------- | ------- |
+| `name` | varchar | 255 | name for project | PK |
+| `description` | longtext | | description of the project | |
+| `created_at` | datetime | 3 | created time of project | |
+| `updated_at` | datetime | 3 | last updated time of project | |
+
+
+| **name** | **describe** | **created_at** | **updated_at** |
+| --------- | ------------------------------------ | ----------------------- | ------------------------|
+| project_1 | this is one of the test projects | 2022-11-01 01:22:13.000 | 2022-11-01 02:24:15.000 |
+| project_2 | this is another project test project | 2022-11-01 01:23:29.000 | 2022-11-01 02:27:24.000 |
+
+## project_metric_settings
+
+| **field** | **type** | **length** | **description** | **key** |
+| --------------- | -------- | ---------- | ---------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `plugin_name` | varchar | 255 | name for plugin | PK |
+| `plugin_option` | longtext | | check if metric plugins have been enabled by the project | |
+| `enable` | tinyint | 1 | if the metric plugins is enabled | |
+
+
+| **project_name** | **plugin_name** | **plugin_option** | **enable** |
+| ---------------- | --------------- | ----------------- | ---------- |
+| project_1 | dora | {} | true |
+| project_2 | dora | {} | false |
+
+## project_mapping
+
+| **field** | **type** | **length** | **description** | **key** |
+| -------------- | -------- | ---------- | ------------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `table` | varchar | 255 | the table name of [Scope](../Overview/KeyConcepts.md#data-scope) | PK |
+| `row_id` | varchar | 255 | the row_id in the [Scope](../Overview/KeyConcepts.md#data-scope) table | PK |
+
+
+| **project_name** | **table** | **row_id** |
+| ---------------- | --------- | ------------------------ |
+| project_1 | Repo | gitlab:GithubRepo:1:lake |
+| project_1 | Board | jira:JiraBoard:1:lake |
+| project_2 | Repo | github:GithubRepo:1:lake |
+
+## _devlake_blueprint
+
+By adding the `project_name` field to the `blueprint` Models, we associate `project` with `blueprint` in one-to-one manner.
+# How to manage project via API
+
+For API specification, please check the swagger doc(by visiting `[Your Config-UI Host]/api/swagger/index.html`).
+Related endpoints:
+
+1. /projects
+2. /projects/:projectName/metrics
+3. /plugins
+
+# The interface that needs to be implemented
+
+As a developer, when we are doing a new plugin development, or when we do some other operations that need to be associated with `Project` to add, delete, and modify operations, we need to have a detailed understanding of `Project`.
+
+We divide plugins into three categories
+- The second category is `Data Source Plugin`, such as `GitLab` `GitHub` `Jira` `Jenkins`, etc. These plugins collect data from various data sources
+- The third category is `Metrics Plugin`, such as `Dora`, etc. These plug-ins do not directly contact the data source but do secondary calculations based on
+
+the collected data after the `Data Source Plugin` works
+
+The second and third types of plug-ins here need to implement the corresponding interface to support the project.
+
+## Data Source Plugin
+
+`Data Source Plugin` is a plugin used to extract data from a data source.
Review Comment:
It is mentioned in line 96 already. no need to repeat
##########
docs/DeveloperManuals/Project.md:
##########
@@ -0,0 +1,184 @@
+---
+title: "Project"
+sidebar_position: 5
+description: >
+ `Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculations like `Dora`.
+---
+
+# Summary
+
+For some metric calculations such as the `DORA` metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.
+
+For example, we may use `Gitlab` for code version control, `Jenkins` for CI/CD, to calculate PR deployment cycle time, we need to know which `Gitlab Projects` and `Jenkins Jobs` are related for correctness and performance reasons.
+
+However, in most cases, we have multiple `Gitlab Projects` / `Jenkins Jobs` that belong to different teams/agendas in our Apache DevLake database.
+
+To distinguish them into different groups. The `Project` is introduced in v0.15. Essentially, a `project` consists of a set of [Scopes](../Overview/KeyConcepts.md#data-scope), i.e., a couple of `Gitlab Projects`, `Jira Boards` or `Jenkins Jobs`, etc.
+
+`Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculation like `Dora`.
+
+Next, let us introduce `Project` in the following order:
+- `Project` related models
+- Related APIs that can be used to manipulate `Project` models
+- The interface that needs to be implemented when developing various plugins to support the `Project`.
+ - The interface that needs to be implemented to develop the `Data Source Plugin`
+ - The interface that needs to be implemented to develop the `Metrics Plugin`
+
+# Models
+
+To support project we contains the following three models:
+ - `projects` describes a project object, including its name, creation and update time and other basic information
+ - `project_metric_settings` describes what metrics plugin a project had enabled.
+ - `project_mapping` describes the mapping relationship of project and scope, including the name of the project、table and row_id.
+
+## projects
+
+| **field** | **type** | **length** | **description** | **key** |
+| ------------- | -------- | ---------- | ----------------------------- | ------- |
+| `name` | varchar | 255 | name for project | PK |
+| `description` | longtext | | description of the project | |
+| `created_at` | datetime | 3 | created time of project | |
+| `updated_at` | datetime | 3 | last updated time of project | |
+
+
+| **name** | **describe** | **created_at** | **updated_at** |
+| --------- | ------------------------------------ | ----------------------- | ------------------------|
+| project_1 | this is one of the test projects | 2022-11-01 01:22:13.000 | 2022-11-01 02:24:15.000 |
+| project_2 | this is another project test project | 2022-11-01 01:23:29.000 | 2022-11-01 02:27:24.000 |
+
+## project_metric_settings
+
+| **field** | **type** | **length** | **description** | **key** |
+| --------------- | -------- | ---------- | ---------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `plugin_name` | varchar | 255 | name for plugin | PK |
+| `plugin_option` | longtext | | check if metric plugins have been enabled by the project | |
+| `enable` | tinyint | 1 | if the metric plugins is enabled | |
+
+
+| **project_name** | **plugin_name** | **plugin_option** | **enable** |
+| ---------------- | --------------- | ----------------- | ---------- |
+| project_1 | dora | {} | true |
+| project_2 | dora | {} | false |
+
+## project_mapping
+
+| **field** | **type** | **length** | **description** | **key** |
+| -------------- | -------- | ---------- | ------------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `table` | varchar | 255 | the table name of [Scope](../Overview/KeyConcepts.md#data-scope) | PK |
+| `row_id` | varchar | 255 | the row_id in the [Scope](../Overview/KeyConcepts.md#data-scope) table | PK |
+
+
+| **project_name** | **table** | **row_id** |
+| ---------------- | --------- | ------------------------ |
+| project_1 | Repo | gitlab:GithubRepo:1:lake |
+| project_1 | Board | jira:JiraBoard:1:lake |
+| project_2 | Repo | github:GithubRepo:1:lake |
+
+## _devlake_blueprint
+
+By adding the `project_name` field to the `blueprint` Models, we associate `project` with `blueprint` in one-to-one manner.
+# How to manage project via API
+
+For API specification, please check the swagger doc(by visiting `[Your Config-UI Host]/api/swagger/index.html`).
+Related endpoints:
+
+1. /projects
+2. /projects/:projectName/metrics
+3. /plugins
+
+# The interface that needs to be implemented
+
+As a developer, when we are doing a new plugin development, or when we do some other operations that need to be associated with `Project` to add, delete, and modify operations, we need to have a detailed understanding of `Project`.
+
+We divide plugins into three categories
+- The second category is `Data Source Plugin`, such as `GitLab` `GitHub` `Jira` `Jenkins`, etc. These plugins collect data from various data sources
Review Comment:
where is the first category?
##########
docs/DeveloperManuals/Project.md:
##########
@@ -0,0 +1,184 @@
+---
+title: "Project"
+sidebar_position: 5
+description: >
+ `Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculations like `Dora`.
+---
+
+# Summary
+
+For some metric calculations such as the `DORA` metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.
+
+For example, we may use `Gitlab` for code version control, `Jenkins` for CI/CD, to calculate PR deployment cycle time, we need to know which `Gitlab Projects` and `Jenkins Jobs` are related for correctness and performance reasons.
+
+However, in most cases, we have multiple `Gitlab Projects` / `Jenkins Jobs` that belong to different teams/agendas in our Apache DevLake database.
+
+To distinguish them into different groups. The `Project` is introduced in v0.15. Essentially, a `project` consists of a set of [Scopes](../Overview/KeyConcepts.md#data-scope), i.e., a couple of `Gitlab Projects`, `Jira Boards` or `Jenkins Jobs`, etc.
+
+`Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculation like `Dora`.
+
+Next, let us introduce `Project` in the following order:
+- `Project` related models
+- Related APIs that can be used to manipulate `Project` models
+- The interface that needs to be implemented when developing various plugins to support the `Project`.
+ - The interface that needs to be implemented to develop the `Data Source Plugin`
+ - The interface that needs to be implemented to develop the `Metrics Plugin`
+
+# Models
+
+To support project we contains the following three models:
+ - `projects` describes a project object, including its name, creation and update time and other basic information
+ - `project_metric_settings` describes what metrics plugin a project had enabled.
+ - `project_mapping` describes the mapping relationship of project and scope, including the name of the project、table and row_id.
+
+## projects
+
+| **field** | **type** | **length** | **description** | **key** |
+| ------------- | -------- | ---------- | ----------------------------- | ------- |
+| `name` | varchar | 255 | name for project | PK |
+| `description` | longtext | | description of the project | |
+| `created_at` | datetime | 3 | created time of project | |
+| `updated_at` | datetime | 3 | last updated time of project | |
+
+
+| **name** | **describe** | **created_at** | **updated_at** |
+| --------- | ------------------------------------ | ----------------------- | ------------------------|
+| project_1 | this is one of the test projects | 2022-11-01 01:22:13.000 | 2022-11-01 02:24:15.000 |
+| project_2 | this is another project test project | 2022-11-01 01:23:29.000 | 2022-11-01 02:27:24.000 |
+
+## project_metric_settings
+
+| **field** | **type** | **length** | **description** | **key** |
+| --------------- | -------- | ---------- | ---------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `plugin_name` | varchar | 255 | name for plugin | PK |
+| `plugin_option` | longtext | | check if metric plugins have been enabled by the project | |
+| `enable` | tinyint | 1 | if the metric plugins is enabled | |
+
+
+| **project_name** | **plugin_name** | **plugin_option** | **enable** |
+| ---------------- | --------------- | ----------------- | ---------- |
+| project_1 | dora | {} | true |
+| project_2 | dora | {} | false |
+
+## project_mapping
+
+| **field** | **type** | **length** | **description** | **key** |
+| -------------- | -------- | ---------- | ------------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `table` | varchar | 255 | the table name of [Scope](../Overview/KeyConcepts.md#data-scope) | PK |
+| `row_id` | varchar | 255 | the row_id in the [Scope](../Overview/KeyConcepts.md#data-scope) table | PK |
+
+
+| **project_name** | **table** | **row_id** |
+| ---------------- | --------- | ------------------------ |
+| project_1 | Repo | gitlab:GithubRepo:1:lake |
+| project_1 | Board | jira:JiraBoard:1:lake |
+| project_2 | Repo | github:GithubRepo:1:lake |
+
+## _devlake_blueprint
+
+By adding the `project_name` field to the `blueprint` Models, we associate `project` with `blueprint` in one-to-one manner.
+# How to manage project via API
+
+For API specification, please check the swagger doc(by visiting `[Your Config-UI Host]/api/swagger/index.html`).
+Related endpoints:
+
+1. /projects
+2. /projects/:projectName/metrics
+3. /plugins
+
+# The interface that needs to be implemented
+
+As a developer, when we are doing a new plugin development, or when we do some other operations that need to be associated with `Project` to add, delete, and modify operations, we need to have a detailed understanding of `Project`.
+
+We divide plugins into three categories
+- The second category is `Data Source Plugin`, such as `GitLab` `GitHub` `Jira` `Jenkins`, etc. These plugins collect data from various data sources
+- The third category is `Metrics Plugin`, such as `Dora`, etc. These plug-ins do not directly contact the data source but do secondary calculations based on
+
+the collected data after the `Data Source Plugin` works
+
+The second and third types of plug-ins here need to implement the corresponding interface to support the project.
+
+## Data Source Plugin
+
+`Data Source Plugin` is a plugin used to extract data from a data source.
+
+For example `GitLab` `GitHub` `Jira` `Jenkins` etc.
+
+These plugins, from various data sources, extract data into the database and store them, they deal directly with the data source, so we classify them as `Data Source Plugin`.
+
+## the DataSourcePluginBlueprintV200 interface
+
+`Data Source Plugin` needs to implement `DataSourcePluginBlueprintV200` interface to support `project`
+
+The interface definition for this interface is as follows
+
+```go
+// DataSourcePluginBlueprintV200 extends the V100 to provide support for
+// Project, so that complex metrics like DORA can be implemented based on a set
+// of Data Scopes
+type DataSourcePluginBlueprintV200 interface {
+ MakeDataSourcePipelinePlanV200(
+ connectionId uint64,
+ scopes []*BlueprintScopeV200,
+ syncPolicy BlueprintSyncPolicy,
+ ) (PipelinePlan, []Scope, errors.Error)
+}
+```
+
+`project` needs to provide a specific set of [Scopes](../Overview/KeyConcepts.md#data-scope) for a specific `connection` to the plug-in through this interface, and then obtain the plug-in involved in the `PipelineTask` All `plugins` and corresponding parameter information. At the same time, the plug-in needs to convert entities like `repo` and `board` in the data source into a `scope interface` that `project` can understand
+
+The corresponding `scope interface` has been implemented at following files of in the framework layer:
+- `models/domainlayer/ticket/board.go`
+- `models/domainlayer/code/repo.go`
Review Comment:
cicd_scope is missing
##########
docs/DeveloperManuals/Project.md:
##########
@@ -0,0 +1,184 @@
+---
+title: "Project"
+sidebar_position: 5
+description: >
+ `Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculations like `Dora`.
+---
+
+# Summary
+
+For some metric calculations such as the `DORA` metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.
+
+For example, we may use `Gitlab` for code version control, `Jenkins` for CI/CD, to calculate PR deployment cycle time, we need to know which `Gitlab Projects` and `Jenkins Jobs` are related for correctness and performance reasons.
+
+However, in most cases, we have multiple `Gitlab Projects` / `Jenkins Jobs` that belong to different teams/agendas in our Apache DevLake database.
+
+To distinguish them into different groups. The `Project` is introduced in v0.15. Essentially, a `project` consists of a set of [Scopes](../Overview/KeyConcepts.md#data-scope), i.e., a couple of `Gitlab Projects`, `Jira Boards` or `Jenkins Jobs`, etc.
+
+`Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculation like `Dora`.
+
+Next, let us introduce `Project` in the following order:
+- `Project` related models
+- Related APIs that can be used to manipulate `Project` models
+- The interface that needs to be implemented when developing various plugins to support the `Project`.
+ - The interface that needs to be implemented to develop the `Data Source Plugin`
+ - The interface that needs to be implemented to develop the `Metrics Plugin`
+
+# Models
+
+To support project we contains the following three models:
+ - `projects` describes a project object, including its name, creation and update time and other basic information
+ - `project_metric_settings` describes what metrics plugin a project had enabled.
+ - `project_mapping` describes the mapping relationship of project and scope, including the name of the project、table and row_id.
+
+## projects
+
+| **field** | **type** | **length** | **description** | **key** |
+| ------------- | -------- | ---------- | ----------------------------- | ------- |
+| `name` | varchar | 255 | name for project | PK |
+| `description` | longtext | | description of the project | |
+| `created_at` | datetime | 3 | created time of project | |
+| `updated_at` | datetime | 3 | last updated time of project | |
+
+
+| **name** | **describe** | **created_at** | **updated_at** |
+| --------- | ------------------------------------ | ----------------------- | ------------------------|
+| project_1 | this is one of the test projects | 2022-11-01 01:22:13.000 | 2022-11-01 02:24:15.000 |
+| project_2 | this is another project test project | 2022-11-01 01:23:29.000 | 2022-11-01 02:27:24.000 |
+
+## project_metric_settings
+
+| **field** | **type** | **length** | **description** | **key** |
+| --------------- | -------- | ---------- | ---------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `plugin_name` | varchar | 255 | name for plugin | PK |
+| `plugin_option` | longtext | | check if metric plugins have been enabled by the project | |
+| `enable` | tinyint | 1 | if the metric plugins is enabled | |
+
+
+| **project_name** | **plugin_name** | **plugin_option** | **enable** |
+| ---------------- | --------------- | ----------------- | ---------- |
+| project_1 | dora | {} | true |
+| project_2 | dora | {} | false |
+
+## project_mapping
+
+| **field** | **type** | **length** | **description** | **key** |
+| -------------- | -------- | ---------- | ------------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `table` | varchar | 255 | the table name of [Scope](../Overview/KeyConcepts.md#data-scope) | PK |
+| `row_id` | varchar | 255 | the row_id in the [Scope](../Overview/KeyConcepts.md#data-scope) table | PK |
+
+
+| **project_name** | **table** | **row_id** |
+| ---------------- | --------- | ------------------------ |
+| project_1 | Repo | gitlab:GithubRepo:1:lake |
+| project_1 | Board | jira:JiraBoard:1:lake |
+| project_2 | Repo | github:GithubRepo:1:lake |
+
+## _devlake_blueprint
+
+By adding the `project_name` field to the `blueprint` Models, we associate `project` with `blueprint` in one-to-one manner.
+# How to manage project via API
+
+For API specification, please check the swagger doc(by visiting `[Your Config-UI Host]/api/swagger/index.html`).
+Related endpoints:
+
+1. /projects
+2. /projects/:projectName/metrics
+3. /plugins
+
+# The interface that needs to be implemented
+
+As a developer, when we are doing a new plugin development, or when we do some other operations that need to be associated with `Project` to add, delete, and modify operations, we need to have a detailed understanding of `Project`.
+
+We divide plugins into three categories
+- The second category is `Data Source Plugin`, such as `GitLab` `GitHub` `Jira` `Jenkins`, etc. These plugins collect data from various data sources
+- The third category is `Metrics Plugin`, such as `Dora`, etc. These plug-ins do not directly contact the data source but do secondary calculations based on
+
+the collected data after the `Data Source Plugin` works
+
+The second and third types of plug-ins here need to implement the corresponding interface to support the project.
+
+## Data Source Plugin
+
+`Data Source Plugin` is a plugin used to extract data from a data source.
+
+For example `GitLab` `GitHub` `Jira` `Jenkins` etc.
+
+These plugins, from various data sources, extract data into the database and store them, they deal directly with the data source, so we classify them as `Data Source Plugin`.
+
+## the DataSourcePluginBlueprintV200 interface
+
+`Data Source Plugin` needs to implement `DataSourcePluginBlueprintV200` interface to support `project`
+
+The interface definition for this interface is as follows
+
+```go
+// DataSourcePluginBlueprintV200 extends the V100 to provide support for
+// Project, so that complex metrics like DORA can be implemented based on a set
+// of Data Scopes
+type DataSourcePluginBlueprintV200 interface {
+ MakeDataSourcePipelinePlanV200(
+ connectionId uint64,
+ scopes []*BlueprintScopeV200,
+ syncPolicy BlueprintSyncPolicy,
+ ) (PipelinePlan, []Scope, errors.Error)
+}
+```
+
+`project` needs to provide a specific set of [Scopes](../Overview/KeyConcepts.md#data-scope) for a specific `connection` to the plug-in through this interface, and then obtain the plug-in involved in the `PipelineTask` All `plugins` and corresponding parameter information. At the same time, the plug-in needs to convert entities like `repo` and `board` in the data source into a `scope interface` that `project` can understand
+
+The corresponding `scope interface` has been implemented at following files of in the framework layer:
+- `models/domainlayer/ticket/board.go`
+- `models/domainlayer/code/repo.go`
+
+In the `plugins/gitlab/impl/impl.go` file, there is a `Gitlab` plugin implementation of the above interface, which can be used as a sample reference.
Review Comment:
remove **sample**
##########
docs/DeveloperManuals/Project.md:
##########
@@ -0,0 +1,184 @@
+---
+title: "Project"
+sidebar_position: 5
+description: >
+ `Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculations like `Dora`.
+---
+
+# Summary
+
+For some metric calculations such as the `DORA` metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.
+
+For example, we may use `Gitlab` for code version control, `Jenkins` for CI/CD, to calculate PR deployment cycle time, we need to know which `Gitlab Projects` and `Jenkins Jobs` are related for correctness and performance reasons.
+
+However, in most cases, we have multiple `Gitlab Projects` / `Jenkins Jobs` that belong to different teams/agendas in our Apache DevLake database.
+
+To distinguish them into different groups. The `Project` is introduced in v0.15. Essentially, a `project` consists of a set of [Scopes](../Overview/KeyConcepts.md#data-scope), i.e., a couple of `Gitlab Projects`, `Jira Boards` or `Jenkins Jobs`, etc.
+
+`Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculation like `Dora`.
+
+Next, let us introduce `Project` in the following order:
+- `Project` related models
+- Related APIs that can be used to manipulate `Project` models
+- The interface that needs to be implemented when developing various plugins to support the `Project`.
+ - The interface that needs to be implemented to develop the `Data Source Plugin`
+ - The interface that needs to be implemented to develop the `Metrics Plugin`
+
+# Models
+
+To support project we contains the following three models:
+ - `projects` describes a project object, including its name, creation and update time and other basic information
+ - `project_metric_settings` describes what metrics plugin a project had enabled.
+ - `project_mapping` describes the mapping relationship of project and scope, including the name of the project、table and row_id.
+
+## projects
+
+| **field** | **type** | **length** | **description** | **key** |
+| ------------- | -------- | ---------- | ----------------------------- | ------- |
+| `name` | varchar | 255 | name for project | PK |
+| `description` | longtext | | description of the project | |
+| `created_at` | datetime | 3 | created time of project | |
+| `updated_at` | datetime | 3 | last updated time of project | |
+
+
+| **name** | **describe** | **created_at** | **updated_at** |
+| --------- | ------------------------------------ | ----------------------- | ------------------------|
+| project_1 | this is one of the test projects | 2022-11-01 01:22:13.000 | 2022-11-01 02:24:15.000 |
+| project_2 | this is another project test project | 2022-11-01 01:23:29.000 | 2022-11-01 02:27:24.000 |
+
+## project_metric_settings
+
+| **field** | **type** | **length** | **description** | **key** |
+| --------------- | -------- | ---------- | ---------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `plugin_name` | varchar | 255 | name for plugin | PK |
+| `plugin_option` | longtext | | check if metric plugins have been enabled by the project | |
+| `enable` | tinyint | 1 | if the metric plugins is enabled | |
+
+
+| **project_name** | **plugin_name** | **plugin_option** | **enable** |
+| ---------------- | --------------- | ----------------- | ---------- |
+| project_1 | dora | {} | true |
+| project_2 | dora | {} | false |
+
+## project_mapping
+
+| **field** | **type** | **length** | **description** | **key** |
+| -------------- | -------- | ---------- | ------------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `table` | varchar | 255 | the table name of [Scope](../Overview/KeyConcepts.md#data-scope) | PK |
+| `row_id` | varchar | 255 | the row_id in the [Scope](../Overview/KeyConcepts.md#data-scope) table | PK |
+
+
+| **project_name** | **table** | **row_id** |
+| ---------------- | --------- | ------------------------ |
+| project_1 | Repo | gitlab:GithubRepo:1:lake |
+| project_1 | Board | jira:JiraBoard:1:lake |
+| project_2 | Repo | github:GithubRepo:1:lake |
+
+## _devlake_blueprint
+
+By adding the `project_name` field to the `blueprint` Models, we associate `project` with `blueprint` in one-to-one manner.
+# How to manage project via API
+
+For API specification, please check the swagger doc(by visiting `[Your Config-UI Host]/api/swagger/index.html`).
+Related endpoints:
+
+1. /projects
+2. /projects/:projectName/metrics
+3. /plugins
+
+# The interface that needs to be implemented
+
+As a developer, when we are doing a new plugin development, or when we do some other operations that need to be associated with `Project` to add, delete, and modify operations, we need to have a detailed understanding of `Project`.
+
+We divide plugins into three categories
+- The second category is `Data Source Plugin`, such as `GitLab` `GitHub` `Jira` `Jenkins`, etc. These plugins collect data from various data sources
+- The third category is `Metrics Plugin`, such as `Dora`, etc. These plug-ins do not directly contact the data source but do secondary calculations based on
+
+the collected data after the `Data Source Plugin` works
+
+The second and third types of plug-ins here need to implement the corresponding interface to support the project.
+
+## Data Source Plugin
+
+`Data Source Plugin` is a plugin used to extract data from a data source.
+
+For example `GitLab` `GitHub` `Jira` `Jenkins` etc.
+
+These plugins, from various data sources, extract data into the database and store them, they deal directly with the data source, so we classify them as `Data Source Plugin`.
+
+## the DataSourcePluginBlueprintV200 interface
+
+`Data Source Plugin` needs to implement `DataSourcePluginBlueprintV200` interface to support `project`
Review Comment:
same as above
##########
docs/DeveloperManuals/Project.md:
##########
@@ -0,0 +1,184 @@
+---
+title: "Project"
+sidebar_position: 5
+description: >
+ `Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculations like `Dora`.
+---
+
+# Summary
+
+For some metric calculations such as the `DORA` metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.
+
+For example, we may use `Gitlab` for code version control, `Jenkins` for CI/CD, to calculate PR deployment cycle time, we need to know which `Gitlab Projects` and `Jenkins Jobs` are related for correctness and performance reasons.
+
+However, in most cases, we have multiple `Gitlab Projects` / `Jenkins Jobs` that belong to different teams/agendas in our Apache DevLake database.
+
+To distinguish them into different groups. The `Project` is introduced in v0.15. Essentially, a `project` consists of a set of [Scopes](../Overview/KeyConcepts.md#data-scope), i.e., a couple of `Gitlab Projects`, `Jira Boards` or `Jenkins Jobs`, etc.
+
+`Project` is **a set of [Scope](../Overview/KeyConcepts.md#data-scope) from different domains**, a way to group different resources, and it is crucial for some metric calculation like `Dora`.
+
+Next, let us introduce `Project` in the following order:
+- `Project` related models
+- Related APIs that can be used to manipulate `Project` models
+- The interface that needs to be implemented when developing various plugins to support the `Project`.
+ - The interface that needs to be implemented to develop the `Data Source Plugin`
+ - The interface that needs to be implemented to develop the `Metrics Plugin`
+
+# Models
+
+To support project we contains the following three models:
+ - `projects` describes a project object, including its name, creation and update time and other basic information
+ - `project_metric_settings` describes what metrics plugin a project had enabled.
+ - `project_mapping` describes the mapping relationship of project and scope, including the name of the project、table and row_id.
+
+## projects
+
+| **field** | **type** | **length** | **description** | **key** |
+| ------------- | -------- | ---------- | ----------------------------- | ------- |
+| `name` | varchar | 255 | name for project | PK |
+| `description` | longtext | | description of the project | |
+| `created_at` | datetime | 3 | created time of project | |
+| `updated_at` | datetime | 3 | last updated time of project | |
+
+
+| **name** | **describe** | **created_at** | **updated_at** |
+| --------- | ------------------------------------ | ----------------------- | ------------------------|
+| project_1 | this is one of the test projects | 2022-11-01 01:22:13.000 | 2022-11-01 02:24:15.000 |
+| project_2 | this is another project test project | 2022-11-01 01:23:29.000 | 2022-11-01 02:27:24.000 |
+
+## project_metric_settings
+
+| **field** | **type** | **length** | **description** | **key** |
+| --------------- | -------- | ---------- | ---------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `plugin_name` | varchar | 255 | name for plugin | PK |
+| `plugin_option` | longtext | | check if metric plugins have been enabled by the project | |
+| `enable` | tinyint | 1 | if the metric plugins is enabled | |
+
+
+| **project_name** | **plugin_name** | **plugin_option** | **enable** |
+| ---------------- | --------------- | ----------------- | ---------- |
+| project_1 | dora | {} | true |
+| project_2 | dora | {} | false |
+
+## project_mapping
+
+| **field** | **type** | **length** | **description** | **key** |
+| -------------- | -------- | ---------- | ------------------------------------------------------------- | ------- |
+| `project_name` | varchar | 255 | name for project | PK |
+| `table` | varchar | 255 | the table name of [Scope](../Overview/KeyConcepts.md#data-scope) | PK |
+| `row_id` | varchar | 255 | the row_id in the [Scope](../Overview/KeyConcepts.md#data-scope) table | PK |
+
+
+| **project_name** | **table** | **row_id** |
+| ---------------- | --------- | ------------------------ |
+| project_1 | Repo | gitlab:GithubRepo:1:lake |
+| project_1 | Board | jira:JiraBoard:1:lake |
+| project_2 | Repo | github:GithubRepo:1:lake |
+
+## _devlake_blueprint
+
+By adding the `project_name` field to the `blueprint` Models, we associate `project` with `blueprint` in one-to-one manner.
+# How to manage project via API
+
+For API specification, please check the swagger doc(by visiting `[Your Config-UI Host]/api/swagger/index.html`).
+Related endpoints:
+
+1. /projects
+2. /projects/:projectName/metrics
+3. /plugins
+
+# The interface that needs to be implemented
+
+As a developer, when we are doing a new plugin development, or when we do some other operations that need to be associated with `Project` to add, delete, and modify operations, we need to have a detailed understanding of `Project`.
+
+We divide plugins into three categories
+- The second category is `Data Source Plugin`, such as `GitLab` `GitHub` `Jira` `Jenkins`, etc. These plugins collect data from various data sources
+- The third category is `Metrics Plugin`, such as `Dora`, etc. These plug-ins do not directly contact the data source but do secondary calculations based on
+
+the collected data after the `Data Source Plugin` works
+
+The second and third types of plug-ins here need to implement the corresponding interface to support the project.
+
+## Data Source Plugin
+
+`Data Source Plugin` is a plugin used to extract data from a data source.
+
+For example `GitLab` `GitHub` `Jira` `Jenkins` etc.
+
+These plugins, from various data sources, extract data into the database and store them, they deal directly with the data source, so we classify them as `Data Source Plugin`.
+
+## the DataSourcePluginBlueprintV200 interface
+
+`Data Source Plugin` needs to implement `DataSourcePluginBlueprintV200` interface to support `project`
+
+The interface definition for this interface is as follows
+
+```go
+// DataSourcePluginBlueprintV200 extends the V100 to provide support for
+// Project, so that complex metrics like DORA can be implemented based on a set
+// of Data Scopes
+type DataSourcePluginBlueprintV200 interface {
+ MakeDataSourcePipelinePlanV200(
+ connectionId uint64,
+ scopes []*BlueprintScopeV200,
+ syncPolicy BlueprintSyncPolicy,
+ ) (PipelinePlan, []Scope, errors.Error)
+}
+```
+
+`project` needs to provide a specific set of [Scopes](../Overview/KeyConcepts.md#data-scope) for a specific `connection` to the plug-in through this interface, and then obtain the plug-in involved in the `PipelineTask` All `plugins` and corresponding parameter information. At the same time, the plug-in needs to convert entities like `repo` and `board` in the data source into a `scope interface` that `project` can understand
+
+The corresponding `scope interface` has been implemented at following files of in the framework layer:
+- `models/domainlayer/ticket/board.go`
+- `models/domainlayer/code/repo.go`
+
+In the `plugins/gitlab/impl/impl.go` file, there is a `Gitlab` plugin implementation of the above interface, which can be used as a sample reference.
+
+And the `plugins/gitlab/api/blueprint_v200.go` contains implementation details.
+
+The following files contain the models that the relevant implementations depend on for reference:
+- `plugins/gitlab/models/project.go`
+- `plugins/gitlab/models/transformation_rule.go`
+
+## Metrics Plugin
+The `Metrics Plugin` is based on the data collected by the `Data Source Plugin` to do secondary calculations.
+
+For example `Dora`, and `Refdff` plugins belong to the `Metrics Plugin`
+
+These plugins are mainly for calculating various metrics, they do not directly contact the data source, so we classify them as `Metrics Plugin`.
+
+## The PluginMetric Interface
+
+`Metrics Plugin` needs to implement the `PluginMetric` interface to support `project`
+
+The interface definition for this interface looks like this:
+
+```go
+type PluginMetric interface {
+ // returns a list of required data entities and expected features.
+ // [{ "model": "cicd_tasks", "requiredFields": {"column": "type", "execptedValue": "Deployment"}}, ...]
+ RequiredDataEntities() (data []map[string]interface{}, err errors.Error)
+
+ // returns if the metric depends on Project for calculation.
+ // Currently, only dora would return true.
+ IsProjectMetric() bool
+
+ // indicates which plugins must be executed before executing this one.
+ // declare a set of dependencies with this
+ RunAfter() ([]string, errors.Error)
+
+ // returns an empty pointer of the plugin setting struct.
+ // (no concrete usage at this point)
+ Settings() (p interface{})
+}
+
+```
+
+`Project` needs `PluginMetric` to know whether a `Metrics Plugin` is dependent on `project`, and the tables and fields required in its calculation process.
Review Comment:
I think the code above is pretty self-explanatory, not need to say more.
However, a section link to "Refercences" is needed here:

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org