You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "coldgust (via GitHub)" <gi...@apache.org> on 2023/03/28 16:35:13 UTC
[GitHub] [incubator-devlake] coldgust opened a new pull request, #4803: feat(circleci): add circleci plugin
coldgust opened a new pull request, #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803
### Summary
Add circleci plugin.
cicd_scopes corresponding to circleci project
cicd_pipelines corresponding to circleci workflow
cicd_pipeline_commits corresponding to circleci vcs
cicd_tasks corresponding to circleci job
### Does this close any open issues?
Closes #4633
### Screenshots
### Other Information
Any other information that is important to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] coldgust commented on pull request #4803: feat(circleci): add circleci plugin
Posted by "coldgust (via GitHub)" <gi...@apache.org>.
coldgust commented on PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803#issuecomment-1492020262
Hi @warren830, I have joined the wechat group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4803: feat(circleci): add circleci plugin
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803#discussion_r1206229677
##########
backend/plugins/circleci/api/connection.go:
##########
@@ -0,0 +1,154 @@
+/*
Review Comment:
@Startrekzky please share any UI prototype images, if available, on the issue (#4633)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on pull request #4803: feat(circleci): add circleci plugin
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803#issuecomment-1547466040
Hi @keon94 @warren830 , are there any updates about the PR, can we merge it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4803: feat(circleci): add circleci plugin
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803#discussion_r1206229235
##########
backend/plugins/circleci/api/connection.go:
##########
@@ -0,0 +1,154 @@
+/*
Review Comment:
@coldgust
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] warren830 commented on pull request #4803: feat(circleci): add circleci plugin
Posted by "warren830 (via GitHub)" <gi...@apache.org>.
warren830 commented on PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803#issuecomment-1489574078
Great job! @coldgust , I will review this asap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] warren830 commented on pull request #4803: feat(circleci): add circleci plugin
Posted by "warren830 (via GitHub)" <gi...@apache.org>.
warren830 commented on PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803#issuecomment-1491177595
Hi @coldgust , are you already in our wechat group?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh merged pull request #4803: feat(circleci): add circleci plugin
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh merged PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4803: feat(circleci): add circleci plugin
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803#discussion_r1194116979
##########
backend/plugins/circleci/api/blueprint200.go:
##########
@@ -0,0 +1,107 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package api
+
+import (
+ "fmt"
+ "github.com/apache/incubator-devlake/plugins/circleci/models"
+ "time"
+
+ "github.com/apache/incubator-devlake/core/dal"
+ "github.com/apache/incubator-devlake/core/errors"
+ "github.com/apache/incubator-devlake/core/models/domainlayer"
+ "github.com/apache/incubator-devlake/core/models/domainlayer/didgen"
+ "github.com/apache/incubator-devlake/core/models/domainlayer/ticket"
+ "github.com/apache/incubator-devlake/core/plugin"
+ "github.com/apache/incubator-devlake/core/utils"
+ helper "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
+)
+
+func MakeDataSourcePipelinePlanV200(subtaskMetas []plugin.SubTaskMeta, connectionId uint64, bpScopes []*plugin.BlueprintScopeV200, syncPolicy *plugin.BlueprintSyncPolicy) (plugin.PipelinePlan, []plugin.Scope, errors.Error) {
+ plan := make(plugin.PipelinePlan, len(bpScopes))
+ plan, err := makeDataSourcePipelinePlanV200(subtaskMetas, plan, bpScopes, connectionId, syncPolicy)
+ if err != nil {
+ return nil, nil, err
+ }
+ scopes, err := makeScopesV200(bpScopes, connectionId)
+ if err != nil {
+ return nil, nil, err
+ }
+
+ return plan, scopes, nil
+}
+
+func makeDataSourcePipelinePlanV200(
+ subtaskMetas []plugin.SubTaskMeta,
+ plan plugin.PipelinePlan,
+ bpScopes []*plugin.BlueprintScopeV200,
+ connectionId uint64,
+ syncPolicy *plugin.BlueprintSyncPolicy,
+) (plugin.PipelinePlan, errors.Error) {
+ for i, bpScope := range bpScopes {
+ stage := plan[i]
+ if stage == nil {
+ stage = plugin.PipelineStage{}
+ }
+ // construct task options for circleci
+ options := make(map[string]interface{})
+ options["projectSlug"] = bpScope.Id
+ options["connectionId"] = connectionId
+ if syncPolicy.TimeAfter != nil {
+ options["timeAfter"] = syncPolicy.TimeAfter.Format(time.RFC3339)
+ }
+
+ subtasks, err := helper.MakePipelinePlanSubtasks(subtaskMetas, bpScope.Entities)
+ if err != nil {
+ return nil, err
+ }
+ stage = append(stage, &plugin.PipelineTask{
+ Plugin: "circleci",
+ Subtasks: subtasks,
+ Options: options,
+ })
+ plan[i] = stage
+ }
+
+ return plan, nil
+}
+
+func makeScopesV200(bpScopes []*plugin.BlueprintScopeV200, connectionId uint64) ([]plugin.Scope, errors.Error) {
+ scopes := make([]plugin.Scope, 0)
+ for _, bpScope := range bpScopes {
+ project := &models.CircleciProject{}
+ // get project from db
+ err := basicRes.GetDal().First(project,
+ dal.Where(`connection_id = ? and id = ?`,
+ connectionId, bpScope.Id))
+ if err != nil {
+ return nil, errors.Default.Wrap(err, fmt.Sprintf("fail to find project %s", bpScope.Id))
+ }
+ // add board to scopes
+ if utils.StringsContains(bpScope.Entities, plugin.DOMAIN_TYPE_TICKET) {
Review Comment:
We need to collect DOMAIN_TYPE_CICD as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4803: feat(circleci): add circleci plugin
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803#discussion_r1156750494
##########
backend/plugins/circleci/api/connection.go:
##########
@@ -0,0 +1,154 @@
+/*
Review Comment:
This will need APIs for scopes, transformation rules and remote_scopes as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4803: feat(circleci): add circleci plugin
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803#discussion_r1194143483
##########
backend/plugins/circleci/api/blueprint200.go:
##########
@@ -0,0 +1,107 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package api
+
+import (
+ "fmt"
+ "github.com/apache/incubator-devlake/plugins/circleci/models"
+ "time"
+
+ "github.com/apache/incubator-devlake/core/dal"
+ "github.com/apache/incubator-devlake/core/errors"
+ "github.com/apache/incubator-devlake/core/models/domainlayer"
+ "github.com/apache/incubator-devlake/core/models/domainlayer/didgen"
+ "github.com/apache/incubator-devlake/core/models/domainlayer/ticket"
+ "github.com/apache/incubator-devlake/core/plugin"
+ "github.com/apache/incubator-devlake/core/utils"
+ helper "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
+)
+
+func MakeDataSourcePipelinePlanV200(subtaskMetas []plugin.SubTaskMeta, connectionId uint64, bpScopes []*plugin.BlueprintScopeV200, syncPolicy *plugin.BlueprintSyncPolicy) (plugin.PipelinePlan, []plugin.Scope, errors.Error) {
+ plan := make(plugin.PipelinePlan, len(bpScopes))
+ plan, err := makeDataSourcePipelinePlanV200(subtaskMetas, plan, bpScopes, connectionId, syncPolicy)
+ if err != nil {
+ return nil, nil, err
+ }
+ scopes, err := makeScopesV200(bpScopes, connectionId)
+ if err != nil {
+ return nil, nil, err
+ }
+
+ return plan, scopes, nil
+}
+
+func makeDataSourcePipelinePlanV200(
+ subtaskMetas []plugin.SubTaskMeta,
+ plan plugin.PipelinePlan,
+ bpScopes []*plugin.BlueprintScopeV200,
+ connectionId uint64,
+ syncPolicy *plugin.BlueprintSyncPolicy,
+) (plugin.PipelinePlan, errors.Error) {
+ for i, bpScope := range bpScopes {
+ stage := plan[i]
+ if stage == nil {
+ stage = plugin.PipelineStage{}
+ }
+ // construct task options for circleci
+ options := make(map[string]interface{})
+ options["projectSlug"] = bpScope.Id
+ options["connectionId"] = connectionId
+ if syncPolicy.TimeAfter != nil {
+ options["timeAfter"] = syncPolicy.TimeAfter.Format(time.RFC3339)
+ }
+
+ subtasks, err := helper.MakePipelinePlanSubtasks(subtaskMetas, bpScope.Entities)
+ if err != nil {
+ return nil, err
+ }
+ stage = append(stage, &plugin.PipelineTask{
+ Plugin: "circleci",
+ Subtasks: subtasks,
+ Options: options,
+ })
+ plan[i] = stage
+ }
+
+ return plan, nil
+}
+
+func makeScopesV200(bpScopes []*plugin.BlueprintScopeV200, connectionId uint64) ([]plugin.Scope, errors.Error) {
+ scopes := make([]plugin.Scope, 0)
+ for _, bpScope := range bpScopes {
+ project := &models.CircleciProject{}
+ // get project from db
+ err := basicRes.GetDal().First(project,
+ dal.Where(`connection_id = ? and id = ?`,
+ connectionId, bpScope.Id))
+ if err != nil {
+ return nil, errors.Default.Wrap(err, fmt.Sprintf("fail to find project %s", bpScope.Id))
+ }
+ // add board to scopes
+ if utils.StringsContains(bpScope.Entities, plugin.DOMAIN_TYPE_TICKET) {
Review Comment:
In fact, it should only be CICD. TICKET won't apply here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] warren830 commented on pull request #4803: feat(circleci): add circleci plugin
Posted by "warren830 (via GitHub)" <gi...@apache.org>.
warren830 commented on PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803#issuecomment-1493311457
>
@coldgust Great, can you send me a msg(you can find me by @nirvana within the group), we want to issue you a certificate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] warren830 commented on pull request #4803: feat(circleci): add circleci plugin
Posted by "warren830 (via GitHub)" <gi...@apache.org>.
warren830 commented on PR #4803:
URL: https://github.com/apache/incubator-devlake/pull/4803#issuecomment-1493322159
@coldgust Still some issues for e2e, please check, thanks/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org