You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by li...@apache.org on 2022/11/16 08:51:57 UTC
[incubator-devlake-website] branch main updated: docs: update plugin implementation and refdiff (#325)
This is an automated email from the ASF dual-hosted git repository.
likyh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
The following commit(s) were added to refs/heads/main by this push:
new 287796f74 docs: update plugin implementation and refdiff (#325)
287796f74 is described below
commit 287796f74799010484d56c293d33f7762ff3e842
Author: Louis.z <lo...@gmail.com>
AuthorDate: Wed Nov 16 16:51:52 2022 +0800
docs: update plugin implementation and refdiff (#325)
Co-authored-by: Startrekzky <ka...@merico.dev>
---
docs/DeveloperManuals/PluginImplementation.md | 22 +++++++++++-----------
docs/Plugins/refdiff.md | 21 +++++++++++----------
2 files changed, 22 insertions(+), 21 deletions(-)
diff --git a/docs/DeveloperManuals/PluginImplementation.md b/docs/DeveloperManuals/PluginImplementation.md
index 34f85ef0e..e1cf04cec 100644
--- a/docs/DeveloperManuals/PluginImplementation.md
+++ b/docs/DeveloperManuals/PluginImplementation.md
@@ -36,8 +36,8 @@ A plugin mainly consists of a collection of subtasks that can be executed by Dev
3. [PluginTask](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_task.go) enables a plugin to prepare data prior to subtask execution
4. [PluginApi](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_api.go) lets a plugin exposes some self-defined APIs
5. [PluginMigration](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_migration.go) is where a plugin manages its database migrations
-6. [PluginModel](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_model.go) allows other plugins to get the model information of all database tables of the current plugin through the GetTablesInfo() method.If you need to access Domain Layer Models,please visit [DomainLayerSchema](https://devlake.apache.org/docs/DataModels/DevLakeDomainLayerSchema/)
-7. [PluginBlueprint](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_blueprint.go) is the foundation for Blueprint and Plugin to collaborate and generate a reasonable Pipeline Plan based on User Settings. For example, a user may declare that he wants to collect data from a Github Repo, which implies that not only the issues/prs, but also the git-meta-data including commits history, branches, tags, etc. needs to be collected. In order to do it and do it faster, l [...]
+6. [PluginModel](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_model.go) allows other plugins to get the model information of all database tables of the current plugin through the GetTablesInfo() method. If you need to access Domain Layer Models, please visit [DomainLayerSchema](https://devlake.apache.org/docs/DataModels/DevLakeDomainLayerSchema/)
+7. [PluginBlueprint](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_blueprint.go) is the foundation for Blueprint and Plugin to collaborate and generate a reasonable Pipeline Plan based on User Settings. For example, a user may declare that he/she wants to collect data from a GitHub Repo, which implies that not only the issues and PRs, but also the git-meta-data including commits history, branches, tags, etc. need to be collected. To do it and do it faster, lev [...]
The diagram below shows the control flow of executing a plugin:
@@ -76,7 +76,7 @@ flowchart TD;
```
There's a lot of information in the diagram, but we don't expect you to digest it right away. You can simply use it as a reference when you go through the example below.
-## A step-by-step guide towards your first conventional plugin
+## A step-by-step guide toward your first conventional plugin
In this section, we will describe how to create a data collection plugin from scratch. The data to be collected is the information about all Committers and Contributors of the Apache project, in order to check whether they have signed the CLA. We are going to
@@ -101,19 +101,19 @@ We will focus on demonstrating how to request and cache information about all Co
> - task data [template](https://github.com/apache/incubator-devlake/blob/main/generator/template/plugin/tasks/task_data.go-template)
> - api client [template](https://github.com/apache/incubator-devlake/blob/main/generator/template/plugin/tasks/task_data_with_api_client.go-template)
-Don't worry if you cannot figure out what these concepts mean immediately. We'll explain them one by one later.
+Don't worry if you cannot figure out what these concepts mean immediately. We'll explain them one by one later.
-DevLake provides a generator to create a plugin conveniently. Let's scaffold our new plugin by running `go run generator/main.go create-plugin icla`, which would ask for `with_api_client` and `Endpoint`.
+Apache DevLake provides a generator to create a plugin conveniently. Let's scaffold our new plugin by running `go run generator/main.go create-plugin icla`, which would ask for `with_api_client` and `Endpoint`.
* `with_api_client` is used for choosing if we need to request HTTP APIs by api_client.
* `Endpoint` use in which site we will request, in our case, it should be `https://people.apache.org/`.
-Now we have three files in our plugin. `api_client.go` and `task_data.go` are in subfolder `tasks/`.
+Now we have three files in our plugin. `api_client.go` and `task_data.go` are in the subfolder `tasks/`.
-Have a try to run this plugin by function `main` in `plugin_main.go`. When you see result like this:
+Have a try to run this plugin by function `main` in `plugin_main.go`. When you see results like this:
```
$go run plugins/icla/plugin_main.go
[2022-06-02 18:07:30] INFO failed to create dir logs: mkdir logs: file exists
@@ -123,10 +123,10 @@ invalid ICLA_TOKEN, but ignore this error now
[2022-06-02 18:07:30] INFO [icla] scheduler for api https://people.apache.org/ worker: 25, request: 18000, duration: 1h0m0s
[2022-06-02 18:07:30] INFO [icla] total step: 0
```
-How exciting. It works! The plugin defined and initiated in `plugin_main.go` use some options in `task_data.go`. They are made up as the most straightforward plugin in Apache DevLake, and `api_client.go` will be used in the next step to request HTTP APIs.
+It works! Plugin 'icla' is defined and initiated. This plugin ONLY contains `plugin_main.go` and `task_data.go`, which is the simplest form of a plugin in Apache DevLake. In the next step, we'll show you how to request HTTP APIs by `api_client.go`.
### Step 2: Create a sub-task for data collection
-Before we start, it is helpful to know how collection task is executed:
+Before we start, it is helpful to know how a collection task is executed:
1. First, Apache DevLake would call `plugin_main.PrepareTaskData()` to prepare needed data before any sub-tasks. We need to create an API client here.
2. Then Apache DevLake will call the sub-tasks returned by `plugin_main.SubTaskMetas()`. Sub-task is an independent task to do some job, like requesting API, processing data, etc.
@@ -145,7 +145,7 @@ Let's run `go run generator/main.go create-collector icla committer` and confirm
> - Collector will collect data from HTTP or other data sources, and save the data into the raw layer.
> - Inside the func `SubTaskEntryPoint` of `Collector`, we use `helper.NewApiCollector` to create an object of [ApiCollector](https://github.com/apache/incubator-devlake/blob/main/generator/template/plugin/tasks/api_collector.go-template), then call `execute()` to do the job.
-Now you can notice `data.ApiClient` is initiated in `plugin_main.go/PrepareTaskData.ApiClient`. `PrepareTaskData` create a new `ApiClient`, which is a tool Apache DevLake suggests to request data from HTTP Apis. This tool support some valuable features for HttpApi, like rateLimit, proxy and retry. Of course, if you like, you may use the lib `http` instead, but it will be more tedious.
+Now you can notice `data.ApiClient` is initiated in `plugin_main.go/PrepareTaskData.ApiClient`. `PrepareTaskData` creates a new `ApiClient`, which is a tool Apache DevLake suggests to request data from HTTP Apis. This tool support some valuable features for HttpApi, like rateLimit, proxy and retry. Of course, if you like, you may use the lib `http` instead, but it will be more tedious.
Let's move forward to use it.
@@ -206,7 +206,7 @@ receive data: 272956 /* <- the number means 272956 models received */
We have already collected data from HTTP API and saved them into the DB table `_raw_XXXX`. In this step, we will extract the names of committers from the raw data. As you may infer from the name, raw tables are temporary and not easy to use directly.
-Now Apache DevLake suggests to save data by [gorm](https://gorm.io/docs/index.html), so we will create a model by gorm and add it into `plugin_main.go/AutoMigrate()`.
+Now Apache DevLake suggests saving data by [gorm](https://gorm.io/docs/index.html), so we will create a model by gorm and add it into `plugin_main.go/AutoMigrate()`.
plugins/icla/models/committer.go
```go
diff --git a/docs/Plugins/refdiff.md b/docs/Plugins/refdiff.md
index 93a6cff8d..d97f1c1da 100644
--- a/docs/Plugins/refdiff.md
+++ b/docs/Plugins/refdiff.md
@@ -7,19 +7,19 @@ description: >
## Summary
-For development workload analysis, we often need to know how many commits have been created between 2 releases. This plugin calculates which commits differ between 2 Ref (branch/tag), and the result will be stored back into database for further analysis.
+For development workload analysis, we often need to know how many commits have been created between 2 releases. This plugin calculates which commits differ between 2 Ref (branch/tag), and the result will be stored back into the database for further analysis.
## Important Note
-You need to run gitextractor before the refdiff plugin. The gitextractor plugin should create records in the `refs` table in your DB before this plugin can be run.
+You need to run `gitextractor` before the `refdiff` plugin. The `gitextractor` plugin should create records in the `refs` table in your database before this plugin can be run.
## Configuration
-This is an enrichment plugin based on Domain Layer data, no configuration needed
+This is an enrichment plugin based on the domain layer data, no configuration is needed.
-## How to use
+## How to use refdiff
-In order to trigger the enrichment, you need to insert a new task into your pipeline.
+To trigger the enrichment, you need to insert a new task into your pipeline.
1. Make sure `commits` and `refs` are collected into your database, `refs` table should contain records like following:
```
@@ -35,7 +35,7 @@ In order to trigger the enrichment, you need to insert a new task into your pipe
github:GithubRepo:384111310:refs/tags/v0.6.1 TAG
```
2. If you want to run calculatePrCherryPick, please configure GITHUB_PR_TITLE_PATTERN in .env, you can check the example in .env.example(we have a default value, please make sure your pattern is disclosed by single quotes '')
-3. And then, trigger a pipeline like following, you can also define sub-tasks, calculateRefDiff will calculate commits between two ref, and creatRefBugStats will create a table to show bug list between two ref:
+3. And then, trigger a pipeline like the following format, you can also define sub-tasks, calculateRefDiff will calculate commits between two refs, and creatRefBugStats will create a table to show a bug list between two refs:
```shell
curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
{
@@ -89,7 +89,8 @@ curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
JSON
```
-4. Refdiff has a new subtask `calculateProjectDeploymentCommitsDiff`, which can find related deployments according to the project, and then calculate the commits diff between deployments. Such a function can be used to support the calculation of dora indicators.
+## How to use refdiff in DORA
+RefDiff can be called by the [DORA plugin](https://github.com/apache/incubator-devlake/tree/main/plugins/dora) to support the calculation of [DORA metrics](https://devlake.apache.org/docs/UserManuals/DORA). RefDiff has a subtask called 'calculateProjectDeploymentCommitsDiff'. This subtask takes the `project_name` from task options to calculate the commits diff between two consecutive deployments in this project. That is to say, refdiff will generate the relationship between `deployed com [...]
```shell
curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
@@ -113,7 +114,7 @@ curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
## Development
-This plugin depends on `libgit2`, you need to install version 1.3.0 in order to run and debug this plugin on your local
+This plugin depends on `libgit2`, you need to install version 1.3.0 to run and debug this plugin on your local
machine.
### Linux
@@ -139,7 +140,7 @@ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
```
-#### Troubleshooting (linux)
+#### Troubleshooting (Linux)
> Q: # pkg-config --cflags -- libgit2 Package libgit2 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libgit2.pc` to the PKG_CONFIG_PATH environment variable
@@ -159,7 +160,7 @@ No package 'libgit2' found pkg-config: exit status 1
### MacOS
-NOTE：Do **NOT** install libgit2 via `MacPorts` or `homebrew`, install from source instead.
+NOTE: **Do NOT** install libgit2 via `MacPorts` or `homebrew`, install from source instead.
```
brew install cmake
git clone https://github.com/libgit2/libgit2.git