You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "chenggui53 (via GitHub)" <gi...@apache.org> on 2023/05/31 10:48:51 UTC

[GitHub] [incubator-devlake] chenggui53 opened a new pull request, #4896: refactor: add subtask register

chenggui53 opened a new pull request, #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896

   ### ⚠️ Pre Checklist
   
   > Please complete _ALL_ items in this checklist, and remove before submitting
   
   - [x] I have read through the [Contributing Documentation](https://devlake.apache.org/community/).
   - [x] I have added relevant tests.
   - [x] I have added relevant documentation.
   - [x] I will add labels to the PR, such as `pr-type/bug-fix`, `pr-type/feature-development`, etc.
   
   <!--
   Thanks for submitting a pull request!
   
   We appreciate you spending the time to work on these changes.
   Please fill out as many sections below as possible.
   -->
   
   ### Summary
   This PR addresses the issue of subtask registration in our GitLab plugin. Previously, subtasks had to be manually specified in a particular order, which was error-prone and time-consuming. To solve this problem, I added a register and dependency sorter that automatically register and orders subtasks based on their dependencies.
   
   ### Does this close any open issues?
   Closes #4413
   
   ### Screenshots
   Include any relevant screenshots here.
   
   ### Other Information
   Any other information that is important to this PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on pull request #4896: refactor: add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1582084569

   @klesh to you think it's time to apply this register to all plugin, or i should create a new pr to do this
   
   > @chenggui53 Thanks for your contribution, it has been a pleasure working with you.
   
   @klesh Me too, and I really appreciate your guidance :). Do you think it's time to apply this register to other plugins, or I should create a new pr to do that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] klesh commented on a diff in pull request #4896: refactor: add subtask register

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#discussion_r1222552758


##########
backend/helpers/pluginhelper/subTaskMetaSorter/interface.go:
##########
@@ -0,0 +1,24 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package subTaskMetaSorter
+
+import "github.com/apache/incubator-devlake/core/plugin"
+
+type SubTaskMetaSorter interface {
+	Sort() ([]plugin.SubTaskMeta, error)

Review Comment:
   I see, maybe we should change the interface as well, that is another story, we can leave it be at the moment



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] klesh commented on a diff in pull request #4896: refactor: add subtask register

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#discussion_r1220750630


##########
backend/helpers/pluginhelper/subTaskMetaSorter/interface.go:
##########
@@ -0,0 +1,24 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package subTaskMetaSorter
+
+import "github.com/apache/incubator-devlake/core/plugin"
+
+type SubTaskMetaSorter interface {
+	Sort() ([]plugin.SubTaskMeta, error)

Review Comment:
   Pointer would be better



##########
backend/plugins/gitlab/tasks/register_test.go:
##########
@@ -0,0 +1,81 @@
+/*

Review Comment:
   The unit test case should be testing the Algo instead of Actual Biz Logic, do NOT re-use the actual dependency from a plugin, or we have to update the test case whenever we add new subtasks.
   Please move this test case `plugin_helper` alongside `DependencySorter` and fabricate some fake dependency to verify the Algo is working correctly.
   



##########
backend/helpers/pluginhelper/subTaskMetaSorter/dependency.go:
##########
@@ -0,0 +1,118 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package subTaskMetaSorter
+
+import (
+	"fmt"
+	"github.com/apache/incubator-devlake/core/plugin"
+	"sort"
+)
+
+type DependencySorter struct {
+	metas []*plugin.SubTaskMeta
+}
+
+func GetDependencySorter(metas []*plugin.SubTaskMeta) SubTaskMetaSorter {
+	return &DependencySorter{metas: metas}
+}
+
+func (d *DependencySorter) Sort() ([]plugin.SubTaskMeta, error) {
+	return topologicalSort(d.metas)
+}
+
+// stable topological sort
+func topologicalSort(metas []*plugin.SubTaskMeta) ([]plugin.SubTaskMeta, error) {
+	// which state will make a cycle
+	dependenciesMap := make(map[string][]string)
+	nameMetaMap := make(map[string]plugin.SubTaskMeta)
+
+	for _, item := range metas {
+		if item != nil {
+			nameMetaMap[item.Name] = *item
+		}
+
+		if _, ok := dependenciesMap[item.Name]; !ok {
+			if len(item.Dependencies) != 0 {
+				dependenciesMap[item.Name] = make([]string, 0)
+				for _, dependencyItem := range item.Dependencies {
+					dependenciesMap[item.Name] = append(dependenciesMap[item.Name], dependencyItem.Name)
+				}
+			} else {
+				dependenciesMap[item.Name] = make([]string, 0)
+			}
+		} else {
+			return nil, fmt.Errorf("got duplicate subtaskmeta in list")
+		}
+	}
+
+	orderedSubtaskList := make([]plugin.SubTaskMeta, 0)
+	for {
+		if len(dependenciesMap) == 0 {
+			break
+		}
+
+		tmpList := make([]string, 0)
+		for key, item := range dependenciesMap {
+			if len(item) == 0 {
+				tmpList = append(tmpList, key)
+			}
+		}
+		if len(tmpList) == 0 {
+			return nil, fmt.Errorf("got error dependency map %v, please check if exist loop", dependenciesMap)

Review Comment:
   Suggested Error message:  cyclic dependency detected



##########
backend/helpers/pluginhelper/subTaskMetaSorter/dependency.go:
##########
@@ -0,0 +1,118 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package subTaskMetaSorter
+
+import (
+	"fmt"
+	"github.com/apache/incubator-devlake/core/plugin"
+	"sort"
+)
+
+type DependencySorter struct {
+	metas []*plugin.SubTaskMeta
+}
+
+func GetDependencySorter(metas []*plugin.SubTaskMeta) SubTaskMetaSorter {
+	return &DependencySorter{metas: metas}
+}
+
+func (d *DependencySorter) Sort() ([]plugin.SubTaskMeta, error) {
+	return topologicalSort(d.metas)
+}
+
+// stable topological sort
+func topologicalSort(metas []*plugin.SubTaskMeta) ([]plugin.SubTaskMeta, error) {
+	// which state will make a cycle
+	dependenciesMap := make(map[string][]string)
+	nameMetaMap := make(map[string]plugin.SubTaskMeta)

Review Comment:
   It would be better to use pointer `*plugin.SubTaskMeta` to avoid memory copy



##########
backend/helpers/pluginhelper/subTaskMetaSorter/dependency.go:
##########
@@ -0,0 +1,118 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package subTaskMetaSorter
+
+import (
+	"fmt"
+	"github.com/apache/incubator-devlake/core/plugin"
+	"sort"
+)
+
+type DependencySorter struct {
+	metas []*plugin.SubTaskMeta
+}
+
+func GetDependencySorter(metas []*plugin.SubTaskMeta) SubTaskMetaSorter {

Review Comment:
   `NewDependencySorter` would be better since the function creates a new instance on each call.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] klesh commented on pull request #4896: refactor: add subtask register

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1577768846

   @chenggui53 Nice job, will do


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1507868319

   > 
   
   
   
   > not sure if we need this kind of mechanism, if so, is it appropriate to sit it on the task level (organize subtasks in a plugin only), or should we do it on the framework level (organize tasks and their subtasks across all plugins). I would like to hear your thoughts on this.
   
   @klesh I personally think that this registration mechanism should be used at the framework level, but since I am not sure whether this change will be accepted, I only modified the gitlab plugin that I use a lot first. If needed, I can change this to the framework level and adapt it to other plugins.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] klesh commented on pull request #4896: refactor: add subtask register

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1582074890

   @chenggui53 Thanks for your contribution, it has been a pleasure working with you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1504758366

   > @chenggui53 Thanks for your contribution. I don't think this can be merged since we need those subtasks to be executed IN ORDER, however, `init()` func breaks this principle.
   
   @klesh Yes, so when I registered subtask with init, I added pre-dependencies declaration and queued the results when fetching the list in backend/plugins/gitlab/impl/impl.go, which was tested in the order defined by the previous code. At the same time, this part will only be called when the plugin is initialized, the ordering will not affect the performance, and can reduce the problems caused by the wrong order of the plugins or the plugins are not registered. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] klesh merged pull request #4896: refactor: add subtask register

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh merged PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#discussion_r1166193370


##########
backend/plugins/gitlab/tasks/tag_collector.go:
##########
@@ -23,6 +23,13 @@ import (
 	helper "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
 )
 
+func init() {
+	register(SubTaskMetaRegister{
+		Plugin:       &CollectTagMeta,
+		Dependencies: []string{ExtractApiMergeRequestDetailsMeta.Name},

Review Comment:
   looks like Dependencies is always just going to be a single subtask? Is there a reason for this to be a slice?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] klesh commented on pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1510071598

   Great discussion, thanks to @CamilleTeruel @chenggui53 
   I would like to add my 2 cents.
   1. auto-discovery may not be easy in Golang, especially since all our plugins get compiled into a `.so` file, there is no official support for scanning Types/Vars in a dynlib. I think `init` func is acceptable if most of us agreed that putting registration in another file is bad.
   2. dependency over entities is indeed a good idea, however, it is kind of complicated, I would accept tasks-base dependency at the moment. I think they can coexist in the future, developers can choose one or another to achieve the goal. We can start on the plugin level by providing a `helper` function to resolve the dependency and see how it goes.
   3. I don't see the `loop detection` in the implementation, the dependency should be a DAG,  I think we should do it.
   4. Is the sorting algo stable? I would prefer a stable outcome for debugging's sake.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1510563002

   > Great discussion, thanks to @CamilleTeruel @chenggui53 I would like to add my 2 cents.
   > 
   > 1. auto-discovery may not be easy in Golang, especially since all our plugins get compiled into a `.so` file, there is no official support for scanning Types/Vars in a dynlib. I think `init` func is acceptable if most of us agreed that putting registration in another file is bad.
   > 2. dependency over entities is indeed a good idea, however, it is kind of complicated, I would accept tasks-base dependency at the moment. I think they can coexist in the future, developers can choose one or another to achieve the goal. We can start on the plugin level by providing a `helper` function to resolve the dependency and see how it goes.
   > 3. I don't see the `loop detection` in the implementation, the dependency should be a DAG,  I think we should do it.
   > 4. Is the sorting algo stable? I would prefer a stable outcome for debugging's sake.
   
   @klesh Thank you very much for your guidance, these are things that I haven't thought about yet, and I will pay attention to these places when designing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on pull request #4896: refactor: add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1578523037

   hi @klesh I‘ve fix some workflow errors, can you help trigger workflow again ?  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on pull request #4896: refactor: add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1576712690

   > Looking good! Good job. Is it ready yet?
   
   hi @klesh, sorry for taking so long, can you help to review this pr when you have time? It mainly includes the following content (to avoid a large number of modifications in each plugin, first only modify the gitlab plugin, and then apply it to other plugins after the code is ok)
   
   in backend/core
   1. Add the subTaskMetas dependencies field in the subTaskMeta structure
   
   in backend/plugin
   1. Stable topological sorting algorithm and loop detection
   2. The sort interface can be extended to entity-level sorting in the future. Currently, only subtask-level sorting is implemented
   
   In backend/plugins/gitlab, take gitlab as an example
   1. In the impl.go file, loop detection will be performed during package init
   2. Add dependencies definition to each subTaskMeta
   2. Each subtask adds an init function for registration
   3. Added the register.go file to register subtask, and added tests to ensure compliance with the tone


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on a diff in pull request #4896: refactor: add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on code in PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#discussion_r1221433286


##########
backend/helpers/pluginhelper/subTaskMetaSorter/interface.go:
##########
@@ -0,0 +1,24 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package subTaskMetaSorter
+
+import "github.com/apache/incubator-devlake/core/plugin"
+
+type SubTaskMetaSorter interface {
+	Sort() ([]plugin.SubTaskMeta, error)

Review Comment:
   @klesh this function can't use pointer cause gitlab/impl/impl.go/subTaskMetas() func needs []plugin.SubtaskMeta return. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] klesh commented on pull request #4896: refactor: add subtask register

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1573001456

   Looking good! Good job. Is it ready yet?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 closed pull request #4896: refactor: add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 closed pull request #4896: refactor: add subtask register
URL: https://github.com/apache/incubator-devlake/pull/4896


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] klesh commented on a diff in pull request #4896: refactor: add subtask register

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#discussion_r1222551659


##########
backend/plugins/gitlab/tasks/register_test.go:
##########
@@ -0,0 +1,81 @@
+/*

Review Comment:
   But the whole point of this PR is to allow developers to add subtasks without worrying about where they should be inserted, right?
   
   If we keep the test case, others would also have to figure out the correct order of given subtasks, even worst, they have to figure out the exact order from the `DependencySorter`, this is harder than NOT using a Sorter, isn't it? What would I do? I would run the Sorter and copy the result to the test case, that is, somewhat pointless...
   
   PS, I notice the folder name `subTaskMetaSorter` is camelCase, can we change it to snake_case? because we are using this convention for the project.❤️



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1504768903

   > @chenggui53 Thanks for your contribution. I don't think this can be merged since we need those subtasks to be executed IN ORDER, however, `init()` func breaks this principle.
   
   @klesh In the short time I've been using the gitlab plugin, I've found some problems caused by manually filling in subtasks (once a specific subtask not being added to the list and not taking effect after being declared in the advanced configuration, and once a subtask being added repeatedly), I think it's worth taking some time to solve it systematically. If Gitlab's plugin registration can be well optimized, the relevant code can also be used in plugins with more subtasks such as GitHub, reducing the possibility of problems.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on a diff in pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on code in PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#discussion_r1166214527


##########
backend/plugins/gitlab/tasks/tag_collector.go:
##########
@@ -23,6 +23,13 @@ import (
 	helper "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
 )
 
+func init() {
+	register(SubTaskMetaRegister{
+		Plugin:       &CollectTagMeta,
+		Dependencies: []string{ExtractApiMergeRequestDetailsMeta.Name},

Review Comment:
   As it stands, we only need to specify a dependent subtask, but if it has dependencies on the results of several subtasks when creating a new subtask, then slice will be useful



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on a diff in pull request #4896: refactor: add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on code in PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#discussion_r1221441248


##########
backend/plugins/gitlab/tasks/register_test.go:
##########
@@ -0,0 +1,81 @@
+/*

Review Comment:
   @klesh i've add algo test in plugin_helper/subTaskMetaSorter dir, and i think the test of DependencySorter should be retain, because 1. Retaining this test allows developers to visually see the execution order of each subtask 2. Developers need to maintain this test to ensure that they understand the dependencies of each subtask in the plug-in (this dependency is not very intuitive when there are many subtasks)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on a diff in pull request #4896: refactor: add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on code in PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#discussion_r1222588403


##########
backend/plugins/gitlab/tasks/register_test.go:
##########
@@ -0,0 +1,81 @@
+/*

Review Comment:
   @klesh you are right, i've delete that test file, maybe someone can find a new way to solve this gracefully :) . The folder name was also changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] CamilleTeruel commented on pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "CamilleTeruel (via GitHub)" <gi...@apache.org>.
CamilleTeruel commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1508529515

   > not sure if we need this kind of mechanism, if so, is it appropriate to sit it on the task level (organize subtasks in a plugin only), or should we do it on the framework level (organize tasks and their subtasks across all plugins). I would like to hear your thoughts on this.
   
   It is currently the responsibility of the plugin to order its subtasks while taking into account their dependencies.
   I think it is valuable to let the plugin declare those dependencies and let the framework order subtasks accordingly. I find it less error-prone because it declares something that was tacit.
   
   Declaring those dependencies might also be a first step to enable other improvements.
   Currently we have a hierarchy of different task types: plans, stages, tasks and subtasks. 
   This has the merit of imposing a structure on task definition but it also comes with costs:
   - Dependencies are implicit: 
       - a subtask *might* depend on a subtask listed before in the same task or on a subtask from an earlier stage. Since we don't know many of the implicit dependencies are artificial and the execution could be parallelized more efficiently. 
       - The plugin developer has to know and remember the execution rules: stages and subtasks are executed sequentially and tasks in parallel. Those execution details could be hidden from the plugins.
   - This make the code to generate pipelines plan complex (~100 lines of procedural code for each plugin)
   - This introduces different protocols for data source plugins and metric plugins. Everything could be declared with dependencies instead.
   - Non-uniformity: there is a possible logic duplication between different types of tasks for concerns like: execution order, resource management, monitoring, caching, error handling, retries, ...
   
   Adding dependency declaration at the subtask level does not solves all those point but is a step in the good direction IMO.
   
   Now regarding implementation, adding a `Dependencies` field in SubtaskMeta looks like a better fit to me than calling a register function in `init` functions. I understand that the purpose of the subtask center is to handle subtask registration, and that calling a function to register each subtask is needed for that. But I think that registration should be handled separately with auto-discovery of subtasks by the framework.
   
   For how to express dependencies, I think we should not force to declare that a subtask depends on another specific subtask but instead declare that the task require that some type of entity has been collected. I agree that it poses no problem for dependencies between subtasks defined by the same plugin: an extractor will always depend on the same convertor for example. But this doesn't work for dependencies between subtasks from different different plugins. For example, the DORA plugin requires some data to be present, like CICDPipelineCommit or PullRequest, but doesn't and mustn't care which plugin collected that data. So the declaration of dependencies should say "I need pull requests" not "I depend on gitlab plugin". I think that declaring dependencies that way would allow us to simplify plugin code and the plugin model of the framework.
   
   To sum up, here is what I think we sould do:
   - Add `Requirements` and `Provisions` slice fields to SubtaskMeta
       - The type of each item is not clear as it should be a supertype of raw data, tool models and domain models. So some refactorings are needed to subsume all those type if we don't want to end up with `[]interface{}` or `[]string` type
       - domain types may be deduced from provisions, so we can later remove the `DomainTypes` field
   - Let the framework order the subtasks of a plugin according to their requirements and provisions
   - Add auto-discovery of a plugin subtasks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] klesh commented on pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1504550378

   @chenggui53 Thanks for your contribution.
   I don't think this can be merged since we need those subtasks to be executed IN ORDER, however, `init()` func breaks this principle.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] klesh commented on pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1507856441

   not sure if we need this kind of mechanism, if so, is it appropriate to sit it on the task level (organize subtasks in a plugin only), or should we do it on the framework level (organize tasks and their subtasks across all plugins). I would like to hear your thoughts on this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] klesh commented on pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1506976545

   @chenggui53 Aah, you are right, sorry for missing the point.
   @keon94 @CamilleTeruel @hezyin @likyh @mindlesscloud what do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake] chenggui53 commented on pull request #4896: refactor(gitlabPlugin): add subtask register

Posted by "chenggui53 (via GitHub)" <gi...@apache.org>.
chenggui53 commented on PR #4896:
URL: https://github.com/apache/incubator-devlake/pull/4896#issuecomment-1509747620

   @CamilleTeruel Thank you very much for your guidance, as I am not particularly familiar with many aspects of devlake, and after reading your instructions, I feel very helpful and I will refer to your instructions to optimize these a bit, which may take some time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org