You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by GitBox <gi...@apache.org> on 2022/11/15 14:47:02 UTC
[GitHub] [incubator-devlake] warren830 opened a new pull request, #3742: feat(dora): update logic for change lead time
warren830 opened a new pull request, #3742:
URL: https://github.com/apache/incubator-devlake/pull/3742
### Summary
In this pr, we update the logic to calculate change lead time
1. update migration script to add table project_pr_metrics, and drop column `repo` for cicd_pipeline_commits(this is a minor bug)
2. add projectName to doraOption
3. use projectName define pr's data scope in dora's task `calculateChangeLeadTime`
4. use the new logic to calculate pr merge time according to #3516
### Does this close any open issues?
closes #3516
### Screenshots
Include any relevant screenshots here.
### Other Information
Any other information that is important to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on a diff in pull request #3742: feat(dora): update logic for change lead time
Posted by GitBox <gi...@apache.org>.
klesh commented on code in PR #3742:
URL: https://github.com/apache/incubator-devlake/pull/3742#discussion_r1024815304
##########
plugins/dora/tasks/change_lead_time_calculator.go:
##########
@@ -34,21 +34,57 @@ import (
func CalculateChangeLeadTime(taskCtx core.SubTaskContext) errors.Error {
db := taskCtx.GetDal()
log := taskCtx.GetLogger()
+ data := taskCtx.GetData().(*DoraTaskData)
+ // construct a list of tuple[task, oldPipelineCommitSha, newPipelineCommitSha, taskFinishedDate]
+ pipelineIdClauses := []dal.Clause{
+ dal.Select(`ct.id as task_id, cpc.commit_sha as new_deploy_commit_sha,
+ ct.finished_date as task_finished_date`),
+ dal.From(`cicd_tasks ct`),
+ dal.Join(`left join cicd_pipeline_commits cpc on ct.pipeline_id = cpc.pipeline_id`),
+ dal.Join(`left join project_mapping pm on pm.row_id = ct.cicd_scope_id and pm.table = 'cicd_scopes'`),
+ dal.Where(`ct.environment = ? and ct.type = ? and pm.project_name = ?`,
Review Comment:
I think we should take `cicd_tasks.status` into consideration.
##########
models/migrationscripts/archived/project_pr_metrics.go:
##########
@@ -0,0 +1,32 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package archived
+
+type ProjectPrMetrics struct {
Review Comment:
please add `deployment_id` to the table as well
##########
plugins/dora/tasks/change_lead_time_calculator.go:
##########
@@ -34,21 +34,57 @@ import (
func CalculateChangeLeadTime(taskCtx core.SubTaskContext) errors.Error {
db := taskCtx.GetDal()
log := taskCtx.GetLogger()
+ data := taskCtx.GetData().(*DoraTaskData)
+ // construct a list of tuple[task, oldPipelineCommitSha, newPipelineCommitSha, taskFinishedDate]
+ pipelineIdClauses := []dal.Clause{
+ dal.Select(`ct.id as task_id, cpc.commit_sha as new_deploy_commit_sha,
+ ct.finished_date as task_finished_date`),
+ dal.From(`cicd_tasks ct`),
+ dal.Join(`left join cicd_pipeline_commits cpc on ct.pipeline_id = cpc.pipeline_id`),
+ dal.Join(`left join project_mapping pm on pm.row_id = ct.cicd_scope_id and pm.table = 'cicd_scopes'`),
+ dal.Where(`ct.environment = ? and ct.type = ? and pm.project_name = ?`,
+ "PRODUCTION", "DEPLOYMENT", data.Options.ProjectName),
+ dal.Orderby(`ct.started_date `),
+ }
+ deploymentPairList := make([]deploymentPair, 0)
+ err := db.All(&deploymentPairList, pipelineIdClauses...)
Review Comment:
Using `pipelineIdClauses` to populate `deploymentPairList` seems odd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] mindlesscloud merged pull request #3742: feat(dora): update logic for change lead time
Posted by GitBox <gi...@apache.org>.
mindlesscloud merged PR #3742:
URL: https://github.com/apache/incubator-devlake/pull/3742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] likyh commented on a diff in pull request #3742: feat(dora): update logic for change lead time
Posted by GitBox <gi...@apache.org>.
likyh commented on code in PR #3742:
URL: https://github.com/apache/incubator-devlake/pull/3742#discussion_r1027878830
##########
plugins/dora/tasks/change_lead_time_calculator.go:
##########
@@ -57,66 +89,74 @@ func CalculateChangeLeadTime(taskCtx core.SubTaskContext) errors.Error {
Input: cursor,
Convert: func(inputRow interface{}) ([]interface{}, errors.Error) {
pr := inputRow.(*code.PullRequest)
- firstCommitDate, err := getFirstCommitTime(pr.Id, db)
+ firstCommit, err := getFirstCommit(pr.Id, db)
+ if err != nil {
+ return nil, err
+ }
+ projectPrMetric := &crossdomain.ProjectPrMetric{}
+ projectPrMetric.Id = pr.Id
+ projectPrMetric.ProjectName = data.Options.ProjectName
if err != nil {
return nil, err
}
- if firstCommitDate != nil {
- codingTime := int64(pr.CreatedDate.Sub(*firstCommitDate).Seconds())
+ if firstCommit != nil {
+ codingTime := int64(pr.CreatedDate.Sub(firstCommit.AuthoredDate).Seconds())
if codingTime/60 == 0 && codingTime%60 > 0 {
Review Comment:
`if codingTime <= 60 {` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] warren830 commented on a diff in pull request #3742: feat(dora): update logic for change lead time
Posted by GitBox <gi...@apache.org>.
warren830 commented on code in PR #3742:
URL: https://github.com/apache/incubator-devlake/pull/3742#discussion_r1023852256
##########
plugins/dora/e2e/raw_tables/project_mapping.csv:
##########
@@ -0,0 +1,4 @@
+project_name,table,row_id,created_at,updated_at,_raw_data_params,_raw_data_table,_raw_data_id,_raw_data_remark
+project1,cicd_scopes,repo2,2019-09-16 14:32:15.000,2015-03-30 03:40:02.000,OKUc0VfGz4,VuOk5CsjZH,155,"Navicat Monitor requires a repository to store alerts and metrics for historical analysis. I may not have gone where I intended to go, but I think I have ended up where I needed to be. To successfully establish a new connection to local/remote server - no matter via SSL, SSH or HTTP, set the database login information in the General tab. What you get by achieving your goals is not as important as what you become by achieving your goals. If you wait, all that happens is you get older. Typically, it is employed as an encrypted version of Telnet. In other words, Navicat provides the ability for data in different databases and/or schemas to be kept up-to-date so that each repository contains the same information. After logged in the Navicat Cloud feature, the Navigation pane will be divided into Navicat Cloud and My Connections sections. Navicat Data Modeler is a powerful and cost-effec
tive database design tool which helps you build high-quality conceptual, logical and physical data models. A man’s best friends are his ten fingers. In a Telnet session, all communications, including username and password, are transmitted in plain-text, allowing anyone to listen-in on your session and steal passwords and other information. You can select any connections, objects or projects, and then select the corresponding buttons on the Information Pane. To get a secure connection, the first thing you need to do is to install OpenSSL Library and download Database Source. There is no way to happiness. Happiness is the way. If you wait, all that happens is you get older. How we spend our days is, of course, how we spend our lives. Navicat provides powerful tools for working with queries: Query Editor for editing the query text directly, and Query Builder, Find Builder or Aggregate Builder for building queries visually. Optimism is the one quality more associated with success and
happiness than any other. There is no way to happiness. Happiness is the way. Monitored servers include MySQL, MariaDB and SQL Server, and compatible with cloud databases like Amazon RDS, Amazon Aurora, Oracle Cloud, Google Cloud and Microsoft Azure. You cannot save people, you can just love them. To get a secure connection, the first thing you need to do is to install OpenSSL Library and download Database Source. Navicat Cloud could not connect and access your databases. By which it means, it could only store your connection settings, queries, model files, and virtual group; your database passwords and data (e.g. tables, views, etc) will not be stored to Navicat Cloud. Creativity is intelligence having fun. What you get by achieving your goals is not as important as what you become by achieving your goals. After comparing data, the window shows the number of records that will be inserted, updated or deleted in the target. Sometimes you win, sometimes you learn. Navicat Cloud could
not connect and access your databases. By which it means, it could only store your connection settings, queries, model files, and virtual group; your database passwords and data (e.g. tables, views, etc) will not be stored to Navicat Cloud. Creativity is intelligence having fun."
Review Comment:
because here cicd_scope_id is equal to repo_id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] warren830 commented on a diff in pull request #3742: feat(dora): update logic for change lead time
Posted by GitBox <gi...@apache.org>.
warren830 commented on code in PR #3742:
URL: https://github.com/apache/incubator-devlake/pull/3742#discussion_r1023861919
##########
plugins/dora/e2e/raw_tables/pull_requests.csv:
##########
@@ -1,10 +1,10 @@
-id,created_at,updated_at,_raw_data_params,_raw_data_table,_raw_data_id,_raw_data_remark,base_repo_id,head_repo_id,status,title,description,url,author_name,author_id,parent_pr_id,pull_request_key,created_date,merged_date,closed_date,type,component,merge_commit_sha,head_ref,base_ref,base_commit_sha,head_commit_sha,coding_timespan,review_lag,review_timespan,deploy_timespan,change_timespan,orig_coding_timespan,orig_review_lag,orig_review_timespan,orig_deploy_timespan
-github:GithubPullRequest:1:1043463302,2022-09-15 05:36:39.856,2022-09-15 05:36:39.856,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,13176,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,feat: implement API for plugin `customize`,"""# Summary\r\n\r\nfulfill the requirement \r\n#2880 ([Feature][customize] implement API for plugin customize) \r\n#2985 ([Feature][customize] new sub-task ExtractCustomizedFields for plugin customize)\r\n\r\nrelated to #2802 \r\nWe take Jira issue as an example.\r\n\r\n1. find all primary keys of table `issues`, which is `id`\r\n2. retrieve data from `issues` and ` _raw_jira_api_issues`\r\n```SQL\r\nSELECT issues._raw_data_id, _raw_jira_api_issues.data, issues.id FROM `issues` LEFT JOIN _raw_jira_api_issues ON issues._raw_data_id = _raw_jira_api_issues.id WHERE _raw_data_table = '_raw_jira_api_issues' AND _raw_data_params = '{\""ConnectionId\"":1,\""BoardId\"":8}'\r\n```\r\n3.
extract values from `_raw_jira_api_issues.data` \r\n4. assign them to the columns in `issues`\r\n\r\n### Does this close any open issues?\r\nCloses #2880 \r\nCloses #2985 \r\n\r\n### Screenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAn example of `TaskOptions`\r\n```json\r\n{\r\n \""transformationRules\"":[ \r\n {\r\n \""table\"":\""issues\"", // domain layer table name\r\n \""rawDataTable\"":\""_raw_jira_api_issues\"", // raw layer table, from which we extract values by json path\r\n \""rawDataParams\"":\""{\\\""ConnectionId\\\"":1,\\\""BoardId\\\"":8}\"", // the value should be a string\r\n \""mapping\"":{\r\n \""x_test\"":\""fields.created\"" // the key is extension field name, the value is json path\r\n }\r\n }\r\n ]\r\n}\r\n```""",https://github.com/apache/incubator-devlake/pull/2911,mindlesscloud,github:GithubAccount:1:8455907,,2911,2022-09-01 09:34:40,2022-09-09 07:52:53,20
22-09-09 07:52:54,,,075e3d3b480e407aede5a0893bd61de9b0fd051d,customize-plugin,main,0006e8105d70318aff5eeee38d405fa181a32aa0,a57007bfeda7b6a15f2ce7901422ca6c8cdc8ede,,,,,,0,0,0,0
-github:GithubPullRequest:1:1048233599,2022-09-15 05:36:39.856,2022-09-15 05:36:39.856,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,13211,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,feat: new plugin for gitea,"""# Summary\r\n[Gitea](https://gitea.io/) is an open-source software package for hosting software development version control using Git as well as other collaborative features like bug tracking, code review, kanban boards, tickets, and wikis.\r\n<!--\r\nThanks for submitting a pull request!\r\n\r\nWe appreciate you spending the time to work on these changes.\r\nPlease fill out as many sections below as possible.\r\n-->\r\n\r\n### Does this close any open issues?\r\nCloses [2992](https://github.com/apache/incubator-devlake/issues/2992)\r\n\r\n### Screenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https:/
/github.com/apache/incubator-devlake/pull/2991,tk103331,github:GithubAccount:1:4404609,,2991,2022-09-07 04:07:39,2022-09-10 02:35:43,2022-09-10 02:35:44,,,3b6c19449a8e7646a31ee2d170df5c55252e5879,gitea-plugin,feat-plugin-gitea,ef013b890f53b98b41611f508554c7084d6b3a28,d715995e019be9ad12c5fed99140303afa967e28,,,,,,0,0,0,0
-github:GithubPullRequest:1:1049191985,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12680,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,[issue-2908]: Bump lake-builder version to 0.0.8,"""### ⚠️ Pre Checklist\r\n\r\n> Please complete _ALL_ items in this checklist, and remove before submitting\r\n\r\n- [x] I have read through the [Contributing Documentation](https://devlake.apache.org/community/).\r\n- [x] I have added relevant tests.\r\n- [x] I have added relevant documentation.\r\n- [ ] I will add labels to the PR, such as `pr-type/bug-fix`, `pr-type/feature-development`, etc.\r\n\r\n\r\n\r\n# Summary\r\n\r\n<!--\r\nThanks for submitting a pull request!\r\n\r\nWe appreciate you spending the time to work on these changes.\r\nPlease fill out as many sections below as possible.\r\n-->\r\n\r\n### Does this close any open issues?\r\nCloses #2908 \r\nBlocks
#2940/#2984\r\nBlocks #2907/#2941\r\n\r\n### Screenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pull/3012,keon94,github:GithubAccount:1:25063936,,3012,2022-09-07 20:30:44,2022-09-09 03:39:50,2022-09-09 03:39:50,,,285c0afd732639d49cc5d0eb7de75c6fa0623cc0,builder-v0.0.8,main,7d0dfb888e2fb5b79cd4654186214e9210e72e98,0edd5d0e0d18516d3309fec90553c3ee7380c149,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051243958,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12691,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,fix: golangci-lint error,"""# Summary\r\n\r\ngolint-ci was down for a while, this PR try to fix all missed linting errors during the period.\r\n\r\n### Screenshots\r\n\r\n\r\n\r\n""",https://github.com/apache/incubator-devlake/pull/3032,klesh,github:GithubAccount:1:61080,,3032,2022-09-09 08:01:07,2022-09-09 17:09:31,2022-09-09 17:09:31,,,ef013b890f53b98b41611f508554c7084d6b3a28,apache,main,fa40bf92015e307900c2f5c03de8404490b5ef4b,62535543802631a0d3daf0b0b78c6a7e05e508fb,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051273681,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12693,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,fix: update pipelineId,"""# Summary\r\n\r\nupdate pipelineId\r\n\r\n### Does this close any open issues?\r\nrelated to #2998 \r\n\r\n### Screenshots\r\n\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pull/3034,abeizn,github:GithubAccount:1:101256042,,3034,2022-09-09 08:31:39,2022-09-09 14:19:14,2022-09-09 14:19:14,,,25b90b1d5e8d9430878f90962c9e8f36ee2c49ca,fix#2998-2,main,075e3d3b480e407aede5a0893bd61de9b0fd051d,209168a4b12d91a7ad1f3711fe05b39d5bd8e260,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051340471,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12694,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,Jenkins fix,"""# Summary\r\nAdd TestJenkinsStagesDataFlow.\r\nAdd path to stages collect.\r\n\r\n<!--\r\nThanks for submitting a pull request!\r\n\r\nWe appreciate you spending the time to work on these changes.\r\nPlease fill out as many sections below as possible.\r\n-->\r\n\r\n### Does this close any open issues?\r\nCloses #3026 \r\n\r\n### Screenshots\r\n\r\n\r\n\r\n\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pul
l/3035,mappjzc,github:GithubAccount:1:2921251,,3035,2022-09-09 09:36:24,2022-09-09 14:47:54,2022-09-09 14:47:54,,,31478092a0afe9bae7de7bdf22e329778564b83f,jenkins-fix,main,075e3d3b480e407aede5a0893bd61de9b0fd051d,08d2f2b6de0fa8de4d0e2b55b4b9a2e244214029,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051524882,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12696,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,fix: task failure caused by 404,"""# Summary\r\nfix #2960 [Bug][gitihub] collect account failed by not found user\r\nIn the case of `err == ErrIgnoreAndContinue`, the `err` should not be wrapped, because on the caller side, the expression `err == ErrIgnoreAndContinue` would evaluate as false.\r\n```go\r\nif apiClient.afterResponse != nil {\r\n\t\terr = apiClient.afterResponse(res)\r\n\t\tif err != nil && err != ErrIgnoreAndContinue {\r\n\t\t\tres.Body.Close()\r\n\t\t\treturn nil, errors.Default.Wrap(err, fmt.Sprintf(\""error running afterRequest for %s\"", req.URL.String()), errors.UserMessage(\""error after making API call\""))\r\n\t\t}\r\n\t}\r\n```\r\n### Does this close any open issues?\r\nCloses #2960 \r\n\r\n### Scr
eenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pull/3038,mindlesscloud,github:GithubAccount:1:8455907,,3038,2022-09-09 12:35:29,2022-09-09 13:32:51,2022-09-09 13:32:51,,,9dc48f963e4f642d385bfe03a60c5d0e33424cb6,ignore-and-continue,main,075e3d3b480e407aede5a0893bd61de9b0fd051d,56b895f0443730c6d7abfbc51a05ab35abd2971f,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051574863,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12697,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,fix(jenkins): update e2e,"""# Summary\r\n\r\nUpdate e2e according to recent changes\r\n\r\n### Does this close any open issues?\r\nrelate to #2854\r\n\r\n\r\n### Screenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pull/3039,warren830,github:GithubAccount:1:39366025,,3039,2022-09-09 13:23:37,2022-09-09 15:12:23,2022-09-09 15:12:23,,,ac0be9ea3a0de2f50246907d33eecb46eaf269ea,fix-update-jenkins-e2e,main,31478092a0afe9bae7de7bdf22e329778564b83f,6fd8efd6b2154da8a0ecf9770d2ac23c8f3111b6,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051637383,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12699,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,fix(gitlab): update e2e,"""# Summary\r\n\r\nupdate gitlab e2e according to recent changes\r\n\r\n### Does this close any open issues?\r\nrelate to #2871\r\n\r\n\r\n### Screenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pull/3041,warren830,github:GithubAccount:1:39366025,,3041,2022-09-09 14:17:24,2022-09-09 15:16:28,2022-09-09 15:16:28,,,fa40bf92015e307900c2f5c03de8404490b5ef4b,fix-update-gitlab-e2e,main,ac0be9ea3a0de2f50246907d33eecb46eaf269ea,9d53fb594958e65456793caa1bfa8d07a7614291,,,,,,0,0,0,0
+id,created_at,updated_at,_raw_data_params,_raw_data_table,_raw_data_id,_raw_data_remark,base_repo_id,head_repo_id,status,title,description,url,author_name,author_id,parent_pr_id,pull_request_key,created_date,merged_date,closed_date,type,component,merge_commit_sha,head_ref,base_ref,base_commit_sha,head_commit_sha
Review Comment:
as we only used merge_date to filter data, so I added some prs with merged_date = null
##########
plugins/dora/e2e/raw_tables/project_mapping.csv:
##########
@@ -0,0 +1,4 @@
+project_name,table,row_id,created_at,updated_at,_raw_data_params,_raw_data_table,_raw_data_id,_raw_data_remark
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] warren830 commented on a diff in pull request #3742: feat(dora): update logic for change lead time
Posted by GitBox <gi...@apache.org>.
warren830 commented on code in PR #3742:
URL: https://github.com/apache/incubator-devlake/pull/3742#discussion_r1023868247
##########
plugins/dora/e2e/raw_tables/commits_diffs.csv:
##########
@@ -0,0 +1,21 @@
+commit_sha,new_commit_sha,old_commit_sha,sorting_index
Review Comment:
fixed, ordered by new_commit_sha, commit_sha
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on a diff in pull request #3742: feat(dora): update logic for change lead time
Posted by GitBox <gi...@apache.org>.
klesh commented on code in PR #3742:
URL: https://github.com/apache/incubator-devlake/pull/3742#discussion_r1023597405
##########
plugins/dora/e2e/raw_tables/cicd_tasks_changeleadtime.csv:
##########
@@ -0,0 +1,19 @@
+id,created_at,updated_at,_raw_data_params,_raw_data_table,_raw_data_id,_raw_data_remark,name,pipeline_id,result,status,type,environment,duration_sec,started_date,finished_date,cicd_scope_id
Review Comment:
Does the lengthy `_raw_data_remark` serve any purpose in the test case? It is hard to review, can we shorten the content of it?
I see
When I view it in LibreOffice, the 6th row didn't display correctly.
The biggest problem is that all `cicd_tasks` are `DEPLOYMENT`, please add other kinds of `cicd_task` between deployment to mimic actually data.
##########
plugins/dora/e2e/raw_tables/commits_diffs.csv:
##########
@@ -0,0 +1,21 @@
+commit_sha,new_commit_sha,old_commit_sha,sorting_index
Review Comment:
Please sort them by `new_commit_sha` / `old_commit_sha` and then `commit_sha` for the sake of the reviewers.
##########
plugins/dora/e2e/raw_tables/project_mapping.csv:
##########
@@ -0,0 +1,4 @@
+project_name,table,row_id,created_at,updated_at,_raw_data_params,_raw_data_table,_raw_data_id,_raw_data_remark
+project1,cicd_scopes,repo2,2019-09-16 14:32:15.000,2015-03-30 03:40:02.000,OKUc0VfGz4,VuOk5CsjZH,155,"Navicat Monitor requires a repository to store alerts and metrics for historical analysis. I may not have gone where I intended to go, but I think I have ended up where I needed to be. To successfully establish a new connection to local/remote server - no matter via SSL, SSH or HTTP, set the database login information in the General tab. What you get by achieving your goals is not as important as what you become by achieving your goals. If you wait, all that happens is you get older. Typically, it is employed as an encrypted version of Telnet. In other words, Navicat provides the ability for data in different databases and/or schemas to be kept up-to-date so that each repository contains the same information. After logged in the Navicat Cloud feature, the Navigation pane will be divided into Navicat Cloud and My Connections sections. Navicat Data Modeler is a powerful and cost-effec
tive database design tool which helps you build high-quality conceptual, logical and physical data models. A man’s best friends are his ten fingers. In a Telnet session, all communications, including username and password, are transmitted in plain-text, allowing anyone to listen-in on your session and steal passwords and other information. You can select any connections, objects or projects, and then select the corresponding buttons on the Information Pane. To get a secure connection, the first thing you need to do is to install OpenSSL Library and download Database Source. There is no way to happiness. Happiness is the way. If you wait, all that happens is you get older. How we spend our days is, of course, how we spend our lives. Navicat provides powerful tools for working with queries: Query Editor for editing the query text directly, and Query Builder, Find Builder or Aggregate Builder for building queries visually. Optimism is the one quality more associated with success and
happiness than any other. There is no way to happiness. Happiness is the way. Monitored servers include MySQL, MariaDB and SQL Server, and compatible with cloud databases like Amazon RDS, Amazon Aurora, Oracle Cloud, Google Cloud and Microsoft Azure. You cannot save people, you can just love them. To get a secure connection, the first thing you need to do is to install OpenSSL Library and download Database Source. Navicat Cloud could not connect and access your databases. By which it means, it could only store your connection settings, queries, model files, and virtual group; your database passwords and data (e.g. tables, views, etc) will not be stored to Navicat Cloud. Creativity is intelligence having fun. What you get by achieving your goals is not as important as what you become by achieving your goals. After comparing data, the window shows the number of records that will be inserted, updated or deleted in the target. Sometimes you win, sometimes you learn. Navicat Cloud could
not connect and access your databases. By which it means, it could only store your connection settings, queries, model files, and virtual group; your database passwords and data (e.g. tables, views, etc) will not be stored to Navicat Cloud. Creativity is intelligence having fun."
Review Comment:
For Change Lead Time calculation, I presume that we first need to filter out all PR by the project first.
How is that possible when the `project1` doesn't contain any record for the `repo`?
##########
plugins/dora/e2e/raw_tables/project_mapping.csv:
##########
@@ -0,0 +1,4 @@
+project_name,table,row_id,created_at,updated_at,_raw_data_params,_raw_data_table,_raw_data_id,_raw_data_remark
Review Comment:
`_raw_xxx` fields are irrelevant in the DORA metric, I think we can omit these fields, same as above.
##########
plugins/dora/e2e/raw_tables/pull_requests.csv:
##########
@@ -1,10 +1,10 @@
-id,created_at,updated_at,_raw_data_params,_raw_data_table,_raw_data_id,_raw_data_remark,base_repo_id,head_repo_id,status,title,description,url,author_name,author_id,parent_pr_id,pull_request_key,created_date,merged_date,closed_date,type,component,merge_commit_sha,head_ref,base_ref,base_commit_sha,head_commit_sha,coding_timespan,review_lag,review_timespan,deploy_timespan,change_timespan,orig_coding_timespan,orig_review_lag,orig_review_timespan,orig_deploy_timespan
-github:GithubPullRequest:1:1043463302,2022-09-15 05:36:39.856,2022-09-15 05:36:39.856,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,13176,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,feat: implement API for plugin `customize`,"""# Summary\r\n\r\nfulfill the requirement \r\n#2880 ([Feature][customize] implement API for plugin customize) \r\n#2985 ([Feature][customize] new sub-task ExtractCustomizedFields for plugin customize)\r\n\r\nrelated to #2802 \r\nWe take Jira issue as an example.\r\n\r\n1. find all primary keys of table `issues`, which is `id`\r\n2. retrieve data from `issues` and ` _raw_jira_api_issues`\r\n```SQL\r\nSELECT issues._raw_data_id, _raw_jira_api_issues.data, issues.id FROM `issues` LEFT JOIN _raw_jira_api_issues ON issues._raw_data_id = _raw_jira_api_issues.id WHERE _raw_data_table = '_raw_jira_api_issues' AND _raw_data_params = '{\""ConnectionId\"":1,\""BoardId\"":8}'\r\n```\r\n3.
extract values from `_raw_jira_api_issues.data` \r\n4. assign them to the columns in `issues`\r\n\r\n### Does this close any open issues?\r\nCloses #2880 \r\nCloses #2985 \r\n\r\n### Screenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAn example of `TaskOptions`\r\n```json\r\n{\r\n \""transformationRules\"":[ \r\n {\r\n \""table\"":\""issues\"", // domain layer table name\r\n \""rawDataTable\"":\""_raw_jira_api_issues\"", // raw layer table, from which we extract values by json path\r\n \""rawDataParams\"":\""{\\\""ConnectionId\\\"":1,\\\""BoardId\\\"":8}\"", // the value should be a string\r\n \""mapping\"":{\r\n \""x_test\"":\""fields.created\"" // the key is extension field name, the value is json path\r\n }\r\n }\r\n ]\r\n}\r\n```""",https://github.com/apache/incubator-devlake/pull/2911,mindlesscloud,github:GithubAccount:1:8455907,,2911,2022-09-01 09:34:40,2022-09-09 07:52:53,20
22-09-09 07:52:54,,,075e3d3b480e407aede5a0893bd61de9b0fd051d,customize-plugin,main,0006e8105d70318aff5eeee38d405fa181a32aa0,a57007bfeda7b6a15f2ce7901422ca6c8cdc8ede,,,,,,0,0,0,0
-github:GithubPullRequest:1:1048233599,2022-09-15 05:36:39.856,2022-09-15 05:36:39.856,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,13211,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,feat: new plugin for gitea,"""# Summary\r\n[Gitea](https://gitea.io/) is an open-source software package for hosting software development version control using Git as well as other collaborative features like bug tracking, code review, kanban boards, tickets, and wikis.\r\n<!--\r\nThanks for submitting a pull request!\r\n\r\nWe appreciate you spending the time to work on these changes.\r\nPlease fill out as many sections below as possible.\r\n-->\r\n\r\n### Does this close any open issues?\r\nCloses [2992](https://github.com/apache/incubator-devlake/issues/2992)\r\n\r\n### Screenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https:/
/github.com/apache/incubator-devlake/pull/2991,tk103331,github:GithubAccount:1:4404609,,2991,2022-09-07 04:07:39,2022-09-10 02:35:43,2022-09-10 02:35:44,,,3b6c19449a8e7646a31ee2d170df5c55252e5879,gitea-plugin,feat-plugin-gitea,ef013b890f53b98b41611f508554c7084d6b3a28,d715995e019be9ad12c5fed99140303afa967e28,,,,,,0,0,0,0
-github:GithubPullRequest:1:1049191985,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12680,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,[issue-2908]: Bump lake-builder version to 0.0.8,"""### ⚠️ Pre Checklist\r\n\r\n> Please complete _ALL_ items in this checklist, and remove before submitting\r\n\r\n- [x] I have read through the [Contributing Documentation](https://devlake.apache.org/community/).\r\n- [x] I have added relevant tests.\r\n- [x] I have added relevant documentation.\r\n- [ ] I will add labels to the PR, such as `pr-type/bug-fix`, `pr-type/feature-development`, etc.\r\n\r\n\r\n\r\n# Summary\r\n\r\n<!--\r\nThanks for submitting a pull request!\r\n\r\nWe appreciate you spending the time to work on these changes.\r\nPlease fill out as many sections below as possible.\r\n-->\r\n\r\n### Does this close any open issues?\r\nCloses #2908 \r\nBlocks
#2940/#2984\r\nBlocks #2907/#2941\r\n\r\n### Screenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pull/3012,keon94,github:GithubAccount:1:25063936,,3012,2022-09-07 20:30:44,2022-09-09 03:39:50,2022-09-09 03:39:50,,,285c0afd732639d49cc5d0eb7de75c6fa0623cc0,builder-v0.0.8,main,7d0dfb888e2fb5b79cd4654186214e9210e72e98,0edd5d0e0d18516d3309fec90553c3ee7380c149,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051243958,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12691,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,fix: golangci-lint error,"""# Summary\r\n\r\ngolint-ci was down for a while, this PR try to fix all missed linting errors during the period.\r\n\r\n### Screenshots\r\n\r\n\r\n\r\n""",https://github.com/apache/incubator-devlake/pull/3032,klesh,github:GithubAccount:1:61080,,3032,2022-09-09 08:01:07,2022-09-09 17:09:31,2022-09-09 17:09:31,,,ef013b890f53b98b41611f508554c7084d6b3a28,apache,main,fa40bf92015e307900c2f5c03de8404490b5ef4b,62535543802631a0d3daf0b0b78c6a7e05e508fb,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051273681,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12693,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,fix: update pipelineId,"""# Summary\r\n\r\nupdate pipelineId\r\n\r\n### Does this close any open issues?\r\nrelated to #2998 \r\n\r\n### Screenshots\r\n\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pull/3034,abeizn,github:GithubAccount:1:101256042,,3034,2022-09-09 08:31:39,2022-09-09 14:19:14,2022-09-09 14:19:14,,,25b90b1d5e8d9430878f90962c9e8f36ee2c49ca,fix#2998-2,main,075e3d3b480e407aede5a0893bd61de9b0fd051d,209168a4b12d91a7ad1f3711fe05b39d5bd8e260,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051340471,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12694,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,Jenkins fix,"""# Summary\r\nAdd TestJenkinsStagesDataFlow.\r\nAdd path to stages collect.\r\n\r\n<!--\r\nThanks for submitting a pull request!\r\n\r\nWe appreciate you spending the time to work on these changes.\r\nPlease fill out as many sections below as possible.\r\n-->\r\n\r\n### Does this close any open issues?\r\nCloses #3026 \r\n\r\n### Screenshots\r\n\r\n\r\n\r\n\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pul
l/3035,mappjzc,github:GithubAccount:1:2921251,,3035,2022-09-09 09:36:24,2022-09-09 14:47:54,2022-09-09 14:47:54,,,31478092a0afe9bae7de7bdf22e329778564b83f,jenkins-fix,main,075e3d3b480e407aede5a0893bd61de9b0fd051d,08d2f2b6de0fa8de4d0e2b55b4b9a2e244214029,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051524882,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12696,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,fix: task failure caused by 404,"""# Summary\r\nfix #2960 [Bug][gitihub] collect account failed by not found user\r\nIn the case of `err == ErrIgnoreAndContinue`, the `err` should not be wrapped, because on the caller side, the expression `err == ErrIgnoreAndContinue` would evaluate as false.\r\n```go\r\nif apiClient.afterResponse != nil {\r\n\t\terr = apiClient.afterResponse(res)\r\n\t\tif err != nil && err != ErrIgnoreAndContinue {\r\n\t\t\tres.Body.Close()\r\n\t\t\treturn nil, errors.Default.Wrap(err, fmt.Sprintf(\""error running afterRequest for %s\"", req.URL.String()), errors.UserMessage(\""error after making API call\""))\r\n\t\t}\r\n\t}\r\n```\r\n### Does this close any open issues?\r\nCloses #2960 \r\n\r\n### Scr
eenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pull/3038,mindlesscloud,github:GithubAccount:1:8455907,,3038,2022-09-09 12:35:29,2022-09-09 13:32:51,2022-09-09 13:32:51,,,9dc48f963e4f642d385bfe03a60c5d0e33424cb6,ignore-and-continue,main,075e3d3b480e407aede5a0893bd61de9b0fd051d,56b895f0443730c6d7abfbc51a05ab35abd2971f,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051574863,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12697,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,fix(jenkins): update e2e,"""# Summary\r\n\r\nUpdate e2e according to recent changes\r\n\r\n### Does this close any open issues?\r\nrelate to #2854\r\n\r\n\r\n### Screenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pull/3039,warren830,github:GithubAccount:1:39366025,,3039,2022-09-09 13:23:37,2022-09-09 15:12:23,2022-09-09 15:12:23,,,ac0be9ea3a0de2f50246907d33eecb46eaf269ea,fix-update-jenkins-e2e,main,31478092a0afe9bae7de7bdf22e329778564b83f,6fd8efd6b2154da8a0ecf9770d2ac23c8f3111b6,,,,,,0,0,0,0
-github:GithubPullRequest:1:1051637383,2022-09-15 05:36:40.170,2022-09-15 05:36:40.170,"{""ConnectionId"":1,""Owner"":""apache"",""Repo"":""incubator-devlake""}",_raw_github_api_pull_requests,12699,,github:GithubRepo:1:384111310,github:GithubRepo:1:384111310,closed,fix(gitlab): update e2e,"""# Summary\r\n\r\nupdate gitlab e2e according to recent changes\r\n\r\n### Does this close any open issues?\r\nrelate to #2871\r\n\r\n\r\n### Screenshots\r\nInclude any relevant screenshots here.\r\n\r\n### Other Information\r\nAny other information that is important to this PR.\r\n""",https://github.com/apache/incubator-devlake/pull/3041,warren830,github:GithubAccount:1:39366025,,3041,2022-09-09 14:17:24,2022-09-09 15:16:28,2022-09-09 15:16:28,,,fa40bf92015e307900c2f5c03de8404490b5ef4b,fix-update-gitlab-e2e,main,ac0be9ea3a0de2f50246907d33eecb46eaf269ea,9d53fb594958e65456793caa1bfa8d07a7614291,,,,,,0,0,0,0
+id,created_at,updated_at,_raw_data_params,_raw_data_table,_raw_data_id,_raw_data_remark,base_repo_id,head_repo_id,status,title,description,url,author_name,author_id,parent_pr_id,pull_request_key,created_date,merged_date,closed_date,type,component,merge_commit_sha,head_ref,base_ref,base_commit_sha,head_commit_sha
Review Comment:
The diversity of the table is far from enough, please add pr that belongs to other repositories, pr with different statuses, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] warren830 commented on pull request #3742: feat(dora): update logic for change lead time
Posted by GitBox <gi...@apache.org>.
warren830 commented on PR #3742:
URL: https://github.com/apache/incubator-devlake/pull/3742#issuecomment-1321949001
> LGTM, and maybe the main logic and other refactor can be split into 2 commits next time. I think it will more clear to know what changed.
Good idea, will try next time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] likyh commented on pull request #3742: feat(dora): update logic for change lead time
Posted by GitBox <gi...@apache.org>.
likyh commented on PR #3742:
URL: https://github.com/apache/incubator-devlake/pull/3742#issuecomment-1321911309
Maybe the main logic and other refactor can be split into 2 commits next time. I think it will more clear to know what changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org