You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by GitBox <gi...@apache.org> on 2022/07/09 14:05:21 UTC
[GitHub] [incubator-devlake] thenicetgp opened a new pull request, #2458: feat: add bitbucket connection and repo
thenicetgp opened a new pull request, #2458:
URL: https://github.com/apache/incubator-devlake/pull/2458
# Summary
add bitbucket connection and repo collector/extractor/convertor
### Does this close any open issues?
partly #2100
### Screenshots
<img width="1402" alt="截屏2022-07-09 21 17 02" src="https://user-images.githubusercontent.com/47466847/178109198-6a8fafeb-f56d-4cb9-a758-7394305ba60
<img width="1402" alt="截屏2022-07-09 21 17 02" src="https://user-images.githubusercontent.com/47466847/178109203-9adb74e5-0db2-45e1-85a7-4d385cdaabc4.png">
5.png">
### Other Information
a draft pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #2458: feat: add a plugin bitbucket
Posted by GitBox <gi...@apache.org>.
keon94 commented on code in PR #2458:
URL: https://github.com/apache/incubator-devlake/pull/2458#discussion_r945943968
##########
plugins/bitbucket/api/blueprint.go:
##########
@@ -0,0 +1,151 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package api
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "io/ioutil"
+ "net/http"
+ "net/url"
+ "strings"
+ "time"
+
+ "github.com/apache/incubator-devlake/models/domainlayer/didgen"
+ "github.com/apache/incubator-devlake/plugins/bitbucket/models"
+ "github.com/apache/incubator-devlake/plugins/bitbucket/tasks"
+ "github.com/apache/incubator-devlake/plugins/core"
+ "github.com/apache/incubator-devlake/plugins/helper"
+ "github.com/apache/incubator-devlake/utils"
+)
+
+func MakePipelinePlan(subtaskMetas []core.SubTaskMeta, connectionId uint64, scope []*core.BlueprintScopeV100) (core.PipelinePlan, error) {
+ var err error
+ plan := make(core.PipelinePlan, len(scope))
+ for i, scopeElem := range scope {
+ // handle taskOptions and transformationRules, by dumping them to taskOptions
+ transformationRules := make(map[string]interface{})
+ if len(scopeElem.Transformation) > 0 {
+ err = json.Unmarshal(scopeElem.Transformation, &transformationRules)
+ if err != nil {
+ return nil, err
+ }
+ }
+ // refdiff
+ if refdiffRules, ok := transformationRules["refdiff"]; ok {
Review Comment:
I'd make "refdiff" a global constant and put it in some shared package. The Github and Gitlab plugins also use it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #2458: feat: add a plugin bitbucket
Posted by GitBox <gi...@apache.org>.
keon94 commented on code in PR #2458:
URL: https://github.com/apache/incubator-devlake/pull/2458#discussion_r945946032
##########
plugins/bitbucket/api/blueprint.go:
##########
@@ -0,0 +1,151 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package api
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "io/ioutil"
+ "net/http"
+ "net/url"
+ "strings"
+ "time"
+
+ "github.com/apache/incubator-devlake/models/domainlayer/didgen"
+ "github.com/apache/incubator-devlake/plugins/bitbucket/models"
+ "github.com/apache/incubator-devlake/plugins/bitbucket/tasks"
+ "github.com/apache/incubator-devlake/plugins/core"
+ "github.com/apache/incubator-devlake/plugins/helper"
+ "github.com/apache/incubator-devlake/utils"
+)
+
+func MakePipelinePlan(subtaskMetas []core.SubTaskMeta, connectionId uint64, scope []*core.BlueprintScopeV100) (core.PipelinePlan, error) {
+ var err error
+ plan := make(core.PipelinePlan, len(scope))
+ for i, scopeElem := range scope {
+ // handle taskOptions and transformationRules, by dumping them to taskOptions
+ transformationRules := make(map[string]interface{})
+ if len(scopeElem.Transformation) > 0 {
+ err = json.Unmarshal(scopeElem.Transformation, &transformationRules)
+ if err != nil {
+ return nil, err
+ }
+ }
+ // refdiff
+ if refdiffRules, ok := transformationRules["refdiff"]; ok {
Review Comment:
more generally, much of this transformation logic seems to be the same as those plugins. Would be good if you can refactor it into a common function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on pull request #2458: feat: add a plugin bitbucket
Posted by GitBox <gi...@apache.org>.
keon94 commented on PR #2458:
URL: https://github.com/apache/incubator-devlake/pull/2458#issuecomment-1215388457
Hi @thenicetgp, could you give an overall status update on this PR? Are there more changes you are working on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] thenicetgp commented on pull request #2458: feat: add a plugin bitbucket
Posted by GitBox <gi...@apache.org>.
thenicetgp commented on PR #2458:
URL: https://github.com/apache/incubator-devlake/pull/2458#issuecomment-1216049042
> Hi @thenicetgp, could you give an overall status update on this PR? Are there more changes you are working on?
Hi @keon94 , thank for your comment, this PR just add a plugin bitbucket, it can currently collect issues,pullrequests,accounts,repos,issue_comments and pr_comments. Domainlayer table accounts,repos,issue,pull_requests,issue_comments and pull_request_comments are filled. These tables have basically all the data they should have.
This PR makes no changes to the any other plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on pull request #2458: feat: add a plugin bitbucket
Posted by GitBox <gi...@apache.org>.
keon94 commented on PR #2458:
URL: https://github.com/apache/incubator-devlake/pull/2458#issuecomment-1194551697
Good work. Make sure you add e2e tests as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] likyh commented on pull request #2458: feat: add bitbucket connection and repo
Posted by GitBox <gi...@apache.org>.
likyh commented on PR #2458:
URL: https://github.com/apache/incubator-devlake/pull/2458#issuecomment-1179555268
Actually, you can convert a pr to a draft pr in the right pannel. 😁
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #2458: feat: add a plugin bitbucket
Posted by GitBox <gi...@apache.org>.
keon94 commented on code in PR #2458:
URL: https://github.com/apache/incubator-devlake/pull/2458#discussion_r945946032
##########
plugins/bitbucket/api/blueprint.go:
##########
@@ -0,0 +1,151 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package api
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "io/ioutil"
+ "net/http"
+ "net/url"
+ "strings"
+ "time"
+
+ "github.com/apache/incubator-devlake/models/domainlayer/didgen"
+ "github.com/apache/incubator-devlake/plugins/bitbucket/models"
+ "github.com/apache/incubator-devlake/plugins/bitbucket/tasks"
+ "github.com/apache/incubator-devlake/plugins/core"
+ "github.com/apache/incubator-devlake/plugins/helper"
+ "github.com/apache/incubator-devlake/utils"
+)
+
+func MakePipelinePlan(subtaskMetas []core.SubTaskMeta, connectionId uint64, scope []*core.BlueprintScopeV100) (core.PipelinePlan, error) {
+ var err error
+ plan := make(core.PipelinePlan, len(scope))
+ for i, scopeElem := range scope {
+ // handle taskOptions and transformationRules, by dumping them to taskOptions
+ transformationRules := make(map[string]interface{})
+ if len(scopeElem.Transformation) > 0 {
+ err = json.Unmarshal(scopeElem.Transformation, &transformationRules)
+ if err != nil {
+ return nil, err
+ }
+ }
+ // refdiff
+ if refdiffRules, ok := transformationRules["refdiff"]; ok {
Review Comment:
more generally, much of this transformation logic seems to be the same as those plugins. Would be good if you can refactor it into a common function. Otherwise, just the global constant will do as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] thenicetgp commented on a diff in pull request #2458: feat: add a plugin bitbucket
Posted by GitBox <gi...@apache.org>.
thenicetgp commented on code in PR #2458:
URL: https://github.com/apache/incubator-devlake/pull/2458#discussion_r946265740
##########
plugins/bitbucket/api/blueprint.go:
##########
@@ -0,0 +1,151 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package api
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "io/ioutil"
+ "net/http"
+ "net/url"
+ "strings"
+ "time"
+
+ "github.com/apache/incubator-devlake/models/domainlayer/didgen"
+ "github.com/apache/incubator-devlake/plugins/bitbucket/models"
+ "github.com/apache/incubator-devlake/plugins/bitbucket/tasks"
+ "github.com/apache/incubator-devlake/plugins/core"
+ "github.com/apache/incubator-devlake/plugins/helper"
+ "github.com/apache/incubator-devlake/utils"
+)
+
+func MakePipelinePlan(subtaskMetas []core.SubTaskMeta, connectionId uint64, scope []*core.BlueprintScopeV100) (core.PipelinePlan, error) {
+ var err error
+ plan := make(core.PipelinePlan, len(scope))
+ for i, scopeElem := range scope {
+ // handle taskOptions and transformationRules, by dumping them to taskOptions
+ transformationRules := make(map[string]interface{})
+ if len(scopeElem.Transformation) > 0 {
+ err = json.Unmarshal(scopeElem.Transformation, &transformationRules)
+ if err != nil {
+ return nil, err
+ }
+ }
+ // refdiff
+ if refdiffRules, ok := transformationRules["refdiff"]; ok {
Review Comment:
got it , thanks for your remind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] thenicetgp commented on pull request #2458: feat: add a plugin bitbucket
Posted by GitBox <gi...@apache.org>.
thenicetgp commented on PR #2458:
URL: https://github.com/apache/incubator-devlake/pull/2458#issuecomment-1195512505
> Good work. Make sure you add e2e tests as well.
thanks,I will add
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] mindlesscloud merged pull request #2458: feat: add a plugin bitbucket
Posted by GitBox <gi...@apache.org>.
mindlesscloud merged PR #2458:
URL: https://github.com/apache/incubator-devlake/pull/2458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org