You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by zk...@apache.org on 2022/12/15 03:50:07 UTC
[incubator-devlake-website] branch main updated: docs: add document for plugin's Incremental and Time Filter (#347)
This is an automated email from the ASF dual-hosted git repository.
zky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
The following commit(s) were added to refs/heads/main by this push:
new 14f35561e docs: add document for plugin's Incremental and Time Filter (#347)
14f35561e is described below
commit 14f35561e34aa9edc4b017b3848198d7ce706c31
Author: Likyh <ya...@meri.co>
AuthorDate: Thu Dec 15 11:50:03 2022 +0800
docs: add document for plugin's Incremental and Time Filter (#347)
* docs: add document for plugin's Incremental and Time Filter
* docs: append
* fix: update for review
* fix: update for review
---
docs/Overview/SupportedDataSources.md | 133 +++++++++++++++++++++++++++++++---
1 file changed, 122 insertions(+), 11 deletions(-)
diff --git a/docs/Overview/SupportedDataSources.md b/docs/Overview/SupportedDataSources.md
index 5f58eb4d2..2cbb17e9e 100644
--- a/docs/Overview/SupportedDataSources.md
+++ b/docs/Overview/SupportedDataSources.md
@@ -9,17 +9,17 @@ sidebar_position: 5
## Data Sources and Data Plugins
Apache DevLake(incubating) supports the following data sources. The data from each data source is collected with one or more plugins.
-| Data Source | Domain(s) | Supported Versions | Config UI Availability | Triggered Plugins | Collection Mode |
-|-------------------------------|------------------------------------------------------------|--------------------------------------|------------------------|---------------------------- | --------------------- |
-| GitHub (include GitHub Action)| Source Code Management, Code Review, Issue Tracking, CI/CD | Cloud |Available |`github`, `gitextractor` | Full Refresh, Incremental Sync(for `issues`, `PRs`) |
-| GitLab (include GitLabCI) | Source Code Management, Code Review, Issue Tracking, CI/CD | Cloud, Community Edition 13.x+ |Available |`gitlab`, `gitextractor` | Full Refresh, Incremental Sync(for `issues`)|
-| Gitee | Source Code Management, Code Review, Issue Tracking | Cloud |Not Available |`gitee`, `gitextractor` | Incremental Sync |
-| BitBucket | Source Code Management, Code Review | Cloud |Not Available |`bitbucket`, `gitextractor` | Full Refresh |
-| Jira | Issue Tracking | Cloud, Server 7.x+, Data Center 7.x+ |Available |`jira` | Full Refresh, Incremental Sync(for `issues`, `changelogs`, `worklogs`) |
-| TAPD | Issue Tracking | Cloud |Not Available |`tapd` | Full Refresh, Incremental Sync(for `stories`, `bugs`, `tasks`) |
-| Jenkins | CI/CD | 2.263.x+ |Available |`jenkins` | Full Refresh |
-| Feishu | Calendar | Cloud |Not Available |`feishu` | Full Refresh |
-| AE | Source Code Management | |Not Available | `ae` | Full Refresh |
+| Data Source | Domain(s) | Supported Versions | Config UI Availability | Triggered Plugins | Collection Mode |
+| ------------------------------ | ---------------------------------------------------------- | ------------------------------------ | ---------------------- | --------------------------- | ------------------------------------------------------------ |
+| GitHub (include GitHub Action) | Source Code Management, Code Review, Issue Tracking, CI/CD | Cloud | Available | `github`, `gitextractor` | Full Refresh |
+| GitLab (include GitLabCI) | Source Code Management, Code Review, Issue Tracking, CI/CD | Cloud, Community Edition 13.x+ | Available | `gitlab`, `gitextractor` | Full Refresh, Incremental Sync(for `issues`,`MRs`) |
+| Gitee | Source Code Management, Code Review, Issue Tracking | Cloud | Not Available | `gitee`, `gitextractor` | Full Refresh, Incremental Sync(for `issues`,`MRs`) |
+| BitBucket | Source Code Management, Code Review | Cloud | Not Available | `bitbucket`, `gitextractor` | Full Refresh |
+| Jira | Issue Tracking | Cloud, Server 7.x+, Data Center 7.x+ | Available | `jira` | Full Refresh, Incremental Sync(for `issues` and related) |
+| TAPD | Issue Tracking | Cloud | Not Available | `tapd` | Full Refresh, Incremental Sync(for `stories`, `bugs`, `tasks`) |
+| Jenkins | CI/CD | 2.263.x+ | Available | `jenkins` | Full Refresh |
+| Feishu | Calendar | Cloud | Not Available | `feishu` | Full Refresh |
+| AE | Source Code Management | | Not Available | `ae` | Full Refresh |
@@ -58,3 +58,114 @@ This table shows the entities collected by each plugin. Domain layer entities in
| builds | | | | | default | | | |
| jobs | | | | | default | | | |
+
+
+
+## Data Sync Policy
+
+**bold:** means it may collect slowly.
+
+**\*bold\*:** means it may collect very slowly.
+
+### Jira
+
+| Subtask Name | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
+| -------------------------- | ------------------------------- | --------------------------------------- | ---------------------------- |
+| CollectStatusMeta | 1 | - | - |
+| CollectProjectsMeta | <10 | ❌ | - |
+| CollectIssueTypesMeta | <10 | ❌ | - |
+| CollectIssuesMeta | <10^4 | ✅ | ✅ |
+| CollectIssueChangelogsMeta | 1000~10^5 | ✅ | ✅ |
+| CollectAccountsMeta | <10^3 | ❌ | ❌ |
+| CollectWorklogsMeta | 1000~10^5 | ✅ | ✅ |
+| CollectRemotelinksMeta | 1000~10^5 | ✅ | ✅ |
+| CollectSprintsMeta | <100 | ❌ | ❌ |
+| CollectEpicsMeta | <100 | ❌ | ✅ |
+
+
+
+### Jenkins
+
+| Subtask Name | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
+| -------------------- | ------------------------------- | --------------------------------------- | ---------------------------- |
+| CollectApiBuildsMeta | ≈100 | ❌ | ❌ |
+| CollectApiStagesMeta | ≈10^4 | ❌ | ✅ |
+
+
+
+### Gitlab
+
+| Subtask Name | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
+| --------------------------- | ------------------------------- | --------------------------------------- | ---------------------------- |
+| CollectApiIssuesMeta | <10^4 | ✅ | ✅ |
+| CollectApiMergeRequestsMeta | <10^3 | ✅ | ✅ |
+| CollectApiMrNotesMeta | <10^5 | ❌ | ✅ |
+| CollectApiMrCommitsMeta | <10^5 | ❌ | ✅ |
+| **CollectApiPipelinesMeta** | <10^4 | ✅ | ❌ |
+| **CollectApiJobsMeta** | <10^5 | ❌ | ❌ |
+
+
+
+### Github
+
+| Subtask Name | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
+| ---------------------------------- | ------------------------------- | --------------------------------------- | ---------------------------- |
+| --------------------------------- | Common | ----------------------- | |
+| CollectMilestonesMeta | ≈10 | ✅ | ❌ |
+| CollectRunsMeta | <10^4 | ❌ | ✅ |
+| CollectApiCommentsMeta | 400 (max page github supported) | ✅ | ❌ |
+| **CollectApiEventsMeta** | 400 (max page github supported) | ❌ | ❌ |
+| CollectApiPullRequestReviewsMeta | <10^5 | ✅ | ✅ |
+| --------------------------------- | Graphql Only (Default) | ----------------------- | |
+| CollectIssueMeta | ≈10^4 | ❌ | ✅ |
+| CollectPrMeta | ≈10^3 | ❌ | ✅ |
+| CollectCheckRunMeta | <10^4 | ❌ | ✅ |
+| CollectAccountMeta | ≈10^2 | ❌ | - |
+| --------------------------------- | Restful Only (Not Default) | ----------------------- | |
+| CollectApiIssuesMeta | ≈10^4 | ✅ | ❌ |
+| CollectApiPullRequestsMeta | ≈10^2 | ✅ | ❌ |
+| CollectApiPullRequestCommitsMeta | ≈10^4 | ✅ | ✅ |
+| **CollectApiPrReviewCommentsMeta** | ≈10^4 | ✅ | ❌ |
+| **CollectAccountsMeta** | ≈10^4 | ❌ | ❌ |
+| **CollectAccountOrgMeta** | ≈10^4 | ❌ | ❌ |
+| **\*CollectJobsMeta\*** | <**10^6** | ❌ | ❌ |
+| CollectApiCommitsMeta | Not enabled | - | - |
+| CollectApiCommitStatsMeta | Not enabled | - | - |
+
+
+
+### Feishu
+
+| Subtask Name | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
+| ----------------------------- | ------------------------------- | --------------------------------------- | ---------------------------- |
+| CollectMeetingTopUserItemMeta | ≈10^3 | ❌ | ✅ |
+
+
+
+### Bitbucket
+
+| Subtask Name | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
+| ----------------------------------- | ------------------------------- | --------------------------------------- | ---------------------------- |
+| ~~CollectApiRepoMeta~~ | 1 | ❌ | ❌ |
+| CollectApiPullRequestsMeta | ≈10^3 | ❌ | ❌ |
+| **CollectApiIssuesMeta** | ≈10^4 | ❌ | ❌ |
+| **CollectApiPrCommentsMeta** | ≈10^5 | ❌ | ❌ |
+| **\*CollectApiIssueCommentsMeta\*** | ≈10^6 | ❌ | ❌ |
+| **CollectApiPipelinesMeta** | <10^4 | ❌ | ❌ |
+| CollectApiDeploymentsMeta | <10^2 | ❌ | ❌ |
+
+
+
+### Gitee
+
+| Subtask Name | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
+| ------------------------------------ | ------------------------------- | --------------------------------------- | ---------------------------- |
+| ~~CollectApiRepoMeta~~ | 1 | ❌ | ❌ |
+| CollectApiPullRequestsMeta | ≈10^3 | ✅ | ❌ |
+| **CollectApiIssuesMeta** | ≈10^4 | ✅ | ❌ |
+| **CollectCommitsMeta?** | ≈10^4 | ✅ | ❌ |
+| **CollectApiPrCommentsMeta** | ≈10^5 | ❌ | ❌ |
+| **\*CollectApiIssueCommentsMeta\*** | ≈10^6 | ✅ | ❌ |
+| **CollectApiPullRequestCommitsMeta** | ≈10^5 | ❌ | ❌ |
+| **CollectApiPullRequestReviewsMeta** | ≈10^5 | ❌ | ❌ |
+| **\*CollectApiCommitStatsMeta\*** | ≈10^6 (Not enable) | ❌ | ❌ |