You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "abeizn (via GitHub)" <gi...@apache.org> on 2023/12/05 10:03:07 UTC
[PR] fix: split the collector into collector+extractor [incubator-devlake]
abeizn opened a new pull request, #6564:
URL: https://github.com/apache/incubator-devlake/pull/6564
### Summary
fix: split the collector into collector+extractor
### Does this close any open issues?
Closes #6495
### Screenshots
Include any relevant screenshots here.
### Other Information
Any other information that is important to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: split the collector into collector+extractor [incubator-devlake]
Posted by "abeizn (via GitHub)" <gi...@apache.org>.
abeizn merged PR #6564:
URL: https://github.com/apache/incubator-devlake/pull/6564
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: split the collector into collector+extractor [incubator-devlake]
Posted by "d4x1 (via GitHub)" <gi...@apache.org>.
d4x1 commented on code in PR #6564:
URL: https://github.com/apache/incubator-devlake/pull/6564#discussion_r1418233886
##########
backend/plugins/github_graphql/tasks/deployment_collector.go:
##########
@@ -130,53 +124,22 @@ func CollectAndExtractDeployments(taskCtx plugin.SubTaskContext) errors.Error {
ResponseParser: func(iQuery interface{}, variables map[string]interface{}) ([]interface{}, error) {
query := iQuery.(*GraphqlQueryDeploymentWrapper)
deployments := query.Repository.Deployments.Deployments
- var results []interface{}
- for _, deployment := range deployments {
- //Skip deployments with createdAt earlier than 'since'
- if deployment.CreatedAt.Before(since.Time) {
- continue
- }
- githubDeployment, err := convertGithubDeployment(deployment, data.Options.ConnectionId, data.Options.GithubId)
- if err != nil {
- return nil, err
+ isFinish := false
Review Comment:
Can `ResponseParser` return `nil, nil` directly?
##########
backend/plugins/github_graphql/tasks/job_extractor.go:
##########
@@ -0,0 +1,106 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package tasks
+
+import (
+ "encoding/json"
+ "strings"
+
+ "github.com/apache/incubator-devlake/core/errors"
+ "github.com/apache/incubator-devlake/core/models/domainlayer/devops"
+ "github.com/apache/incubator-devlake/core/plugin"
+ "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
+ "github.com/apache/incubator-devlake/plugins/github/models"
+ githubTasks "github.com/apache/incubator-devlake/plugins/github/tasks"
+)
+
+var _ plugin.SubTaskEntryPoint = ExtractAccounts
+
+var ExtractJobsMeta = plugin.SubTaskMeta{
+ Name: "extractJobs",
+ EntryPoint: ExtractJobs,
+ EnabledByDefault: true,
+ Description: "Extract raw run data into tool layer table github_jobs",
+ DomainTypes: []string{plugin.DOMAIN_TYPE_CICD},
+}
+
+func ExtractJobs(taskCtx plugin.SubTaskContext) errors.Error {
+ data := taskCtx.GetData().(*githubTasks.GithubTaskData)
+ extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
+ RawDataSubTaskArgs: api.RawDataSubTaskArgs{
+ Ctx: taskCtx,
+ Params: githubTasks.GithubApiParams{
+ ConnectionId: data.Options.ConnectionId,
+ Name: data.Options.Name,
+ },
+ Table: RAW_GRAPHQL_JOBS_TABLE,
+ },
+ Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
+ apiJob := &GraphqlQueryCheckRunWrapper{}
+ err := errors.Convert(json.Unmarshal(row.Data, apiJob))
+ if err != nil {
+ return nil, err
+ }
+
+ nodes := apiJob.Node
+ results := make([]interface{}, 0, 1)
+ for _, node := range nodes {
+ for _, checkRun := range node.CheckSuite.CheckRuns.Nodes {
+
+ paramsBytes, err := json.Marshal(checkRun.Steps.Nodes)
+ if err != nil {
+ taskCtx.GetLogger().Error(err, `Marshal checkRun.Steps.Nodes fail and ignore`)
+ }
+ githubJob := &models.GithubJob{
+ ConnectionId: data.Options.ConnectionId,
+ RunID: node.CheckSuite.WorkflowRun.DatabaseId,
+ RepoId: data.Options.GithubId,
+ ID: checkRun.DatabaseId,
+ NodeID: checkRun.Id,
+ HTMLURL: checkRun.DetailsUrl,
+ Status: strings.ToUpper(checkRun.Status),
+ Conclusion: strings.ToUpper(checkRun.Conclusion),
+ StartedAt: checkRun.StartedAt,
+ CompletedAt: checkRun.CompletedAt,
+ Name: checkRun.Name,
+ Steps: paramsBytes,
+ Type: data.RegexEnricher.ReturnNameIfMatched(devops.DEPLOYMENT, checkRun.Name),
+ Environment: data.RegexEnricher.ReturnNameIfOmittedOrMatched(devops.PRODUCTION, checkRun.Name),
+ // these columns can not fill by graphql
+ //HeadSha: ``, // use _tool_github_runs
+ //RunURL: ``,
+ //CheckRunURL: ``,
+ //Labels: ``, // not on use
+ //RunnerID: ``, // not on use
+ //RunnerName: ``, // not on use
+ //RunnerGroupID: ``, // not on use
Review Comment:
I think `not on use` should be `not in use`.
##########
backend/plugins/github_graphql/tasks/pr_collector.go:
##########
@@ -119,35 +117,19 @@ type GraphqlQueryCommit struct {
Url string
}
-var CollectPrMeta = plugin.SubTaskMeta{
- Name: "CollectPr",
- EntryPoint: CollectPr,
+var CollectPrsMeta = plugin.SubTaskMeta{
+ Name: "CollectPrs",
Review Comment:
Will this name be conflict with tasks in github/tasks?
##########
backend/plugins/github_graphql/tasks/deployment_collector.go:
##########
@@ -85,9 +83,9 @@ type GraphqlQueryDeploymentDeployment struct {
UpdatedAt time.Time
}
-// CollectAndExtractDeployments will request github api via graphql and store the result into raw layer by default
+// CollectDeployments will request github api via graphql and store the result into raw layer by default
// ResponseParser's return will be stored to tool layer. So it's called CollectorAndExtractor.
-func CollectAndExtractDeployments(taskCtx plugin.SubTaskContext) errors.Error {
+func CollectDeployments(taskCtx plugin.SubTaskContext) errors.Error {
Review Comment:
Comment in line 87 should be updated too.
##########
backend/plugins/github_graphql/tasks/deployment_collector.go:
##########
@@ -22,23 +22,21 @@ import (
"time"
"github.com/apache/incubator-devlake/core/errors"
- "github.com/apache/incubator-devlake/core/models/common"
"github.com/apache/incubator-devlake/core/plugin"
helper "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
- githubModels "github.com/apache/incubator-devlake/plugins/github/models"
githubTasks "github.com/apache/incubator-devlake/plugins/github/tasks"
"github.com/merico-dev/graphql"
)
-var _ plugin.SubTaskEntryPoint = CollectAndExtractDeployments
+var _ plugin.SubTaskEntryPoint = CollectDeployments
const (
RAW_DEPLOYMENT = "github_graphql_deployment"
)
-var CollectAndExtractDeploymentsMeta = plugin.SubTaskMeta{
- Name: "CollectAndExtractDeployments",
- EntryPoint: CollectAndExtractDeployments,
+var CollectDeploymentsMeta = plugin.SubTaskMeta{
+ Name: "CollectDeployments",
+ EntryPoint: CollectDeployments,
EnabledByDefault: true,
Description: "collect and extract github deployments to raw and tool layer from GithubGraphql api",
Review Comment:
`Description` should be updated too.
##########
backend/plugins/github_graphql/tasks/account_extractor.go:
##########
@@ -0,0 +1,106 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package tasks
+
+import (
+ "encoding/json"
+
+ "github.com/apache/incubator-devlake/core/errors"
+ "github.com/apache/incubator-devlake/core/plugin"
+ "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
+ "github.com/apache/incubator-devlake/plugins/github/models"
+ githubTasks "github.com/apache/incubator-devlake/plugins/github/tasks"
+)
+
+var _ plugin.SubTaskEntryPoint = ExtractAccounts
+
+var ExtractAccountsMeta = plugin.SubTaskMeta{
+ Name: "extractAccounts",
+ EntryPoint: ExtractAccounts,
+ EnabledByDefault: true,
+ Description: "extract raw account data into tool layer table github_accounts",
+ DomainTypes: []string{plugin.DOMAIN_TYPE_CROSS},
+}
+
+func ExtractAccounts(taskCtx plugin.SubTaskContext) errors.Error {
+ data := taskCtx.GetData().(*githubTasks.GithubTaskData)
+ extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
+ RawDataSubTaskArgs: api.RawDataSubTaskArgs{
+ Ctx: taskCtx,
+ Params: githubTasks.GithubApiParams{
+ ConnectionId: data.Options.ConnectionId,
+ Name: data.Options.Name,
+ },
+ Table: RAW_ACCOUNTS_TABLE,
+ },
+ Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
+ apiAccount := &GraphqlQueryAccountWrapper{}
+ err := errors.Convert(json.Unmarshal(row.Data, apiAccount))
+ if err != nil {
+ return nil, err
+ }
+ accounts := apiAccount.Users
+ results := make([]interface{}, 0, 1)
Review Comment:
`var result []interface` is more idiomatic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: split the collector into collector+extractor [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6564:
URL: https://github.com/apache/incubator-devlake/pull/6564#issuecomment-1845065594
🤖 cherry pick finished successfully 🎉!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: split the collector into collector+extractor [incubator-devlake]
Posted by "abeizn (via GitHub)" <gi...@apache.org>.
abeizn commented on code in PR #6564:
URL: https://github.com/apache/incubator-devlake/pull/6564#discussion_r1418238879
##########
backend/plugins/github_graphql/tasks/pr_collector.go:
##########
@@ -119,35 +117,19 @@ type GraphqlQueryCommit struct {
Url string
}
-var CollectPrMeta = plugin.SubTaskMeta{
- Name: "CollectPr",
- EntryPoint: CollectPr,
+var CollectPrsMeta = plugin.SubTaskMeta{
+ Name: "CollectPrs",
Review Comment:
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org