You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "d4x1 (via GitHub)" <gi...@apache.org> on 2023/12/14 03:21:02 UTC

[PR] Feat 6520 [incubator-devlake]

d4x1 opened a new pull request, #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628

   ### ⚠️ Pre Checklist
   
   > Please complete _ALL_ items in this checklist, and remove before submitting
   
   - [x] I have read through the [Contributing Documentation](https://devlake.apache.org/community/).
   - [x] I have added relevant tests.
   - [x] I have added relevant documentation.
   - [x] I will add labels to the PR, such as `pr-type/bug-fix`, `pr-type/feature-development`, etc.
   
   <!--
   Thanks for submitting a pull request!
   
   We appreciate you spending the time to work on these changes.
   Please fill out as many sections below as possible.
   -->
   
   ### Summary
   What does this PR do?
   1. Add new api that can fetch scope's latest sync state
   
   ### Does this close any open issues?
   Closes #6520 
   
   ### Screenshots
   Include any relevant screenshots here.
   #### GitHub
   <img width="1847" alt="image" src="https://github.com/apache/incubator-devlake/assets/5844806/4453a5bc-a62d-4f7e-817d-eb87ef6f9ac5">
   
   
   ### Other Information
   Any other information that is important to this PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "d4x1 (via GitHub)" <gi...@apache.org>.
d4x1 commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1426697473


##########
backend/helpers/srvhelper/scope_service_helper.go:
##########
@@ -93,6 +93,44 @@ func (scopeSrv *ScopeSrvHelper[C, S, SC]) GetScopeDetail(includeBlueprints bool,
 	return scopeDetail, nil
 }
 
+func (scopeSrv *ScopeSrvHelper[C, S, SC]) GetScopeLatestSyncState(pkv ...interface{}) ([]*models.LatestSyncState, errors.Error) {
+	scope, err := scopeSrv.ModelSrvHelper.FindByPk(pkv...)
+	if err != nil {
+		return nil, err
+	}
+	s := *scope
+	params := plugin.MarshalScopeParams(s.ScopeParams())
+	scopeSrv.log.Debug("scope: %#+v, params: %+v", s, params)
+	rows, err := scopeSrv.db.Cursor(
+		dal.Select("raw_data_table, MAX(latest_success_start) as latest_success_start"),
+		dal.From("_devlake_collector_latest_state"),
+		dal.Where("raw_data_params = ?", params),
+		dal.Groupby("raw_data_table"),
+	)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	scopeSyncStates := []*models.LatestSyncState{}

Review Comment:
   > Seems like we are returning the whole list anyway, I suggest that we use the `All` method instead of using `Cursor` to make it simpler and readable
   
   Got it. I'll fix it later.



##########
backend/helpers/srvhelper/scope_service_helper.go:
##########
@@ -93,6 +93,44 @@ func (scopeSrv *ScopeSrvHelper[C, S, SC]) GetScopeDetail(includeBlueprints bool,
 	return scopeDetail, nil
 }
 
+func (scopeSrv *ScopeSrvHelper[C, S, SC]) GetScopeLatestSyncState(pkv ...interface{}) ([]*models.LatestSyncState, errors.Error) {
+	scope, err := scopeSrv.ModelSrvHelper.FindByPk(pkv...)
+	if err != nil {
+		return nil, err
+	}
+	s := *scope
+	params := plugin.MarshalScopeParams(s.ScopeParams())
+	scopeSrv.log.Debug("scope: %#+v, params: %+v", s, params)
+	rows, err := scopeSrv.db.Cursor(
+		dal.Select("raw_data_table, MAX(latest_success_start) as latest_success_start"),

Review Comment:
   > I don't think group by `raw_data_table` and take the max of `latest_success_start` is needed because `raw_data_table`s are unique when `raw_data_params` is specified. I think simply select the needed field would be sufficient.
   
   Got it. I'll fix it later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "d4x1 (via GitHub)" <gi...@apache.org>.
d4x1 commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1429619427


##########
backend/plugins/bitbucket/api/scope.go:
##########
@@ -82,6 +82,25 @@ func GetScopeList(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, er
 	return scopeHelper.GetScopeList(input)
 }
 
+// GetScopeDispatcher process BitBucket repos get request
+// @Summary process BitBucket repos get request
+// @Description process BitBucket repos get request
+// @Tags plugins/bitbucket
+// @Param connectionId path int true "connection ID"
+// @Param scopeId path string true "repo ID"
+// @Success 200  {object} ScopeRes
+// @Failure 400  {object} shared.ApiBody "Bad Request"
+// @Failure 500  {object} shared.ApiBody "Internal Error"
+// @Router /plugins/bitbucket/connections/{connectionId}/scopes/{scopeId} [GET]
+func GetScopeDispatcher(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {

Review Comment:
   @klesh 
   Do you mean we change all `latest-sync-state` to `/plugins/{plugin_name}/sync_state` in all plugins ? or just bitbucket ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "abeizn (via GitHub)" <gi...@apache.org>.
abeizn merged PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1426491190


##########
backend/helpers/srvhelper/scope_service_helper.go:
##########
@@ -93,6 +93,44 @@ func (scopeSrv *ScopeSrvHelper[C, S, SC]) GetScopeDetail(includeBlueprints bool,
 	return scopeDetail, nil
 }
 
+func (scopeSrv *ScopeSrvHelper[C, S, SC]) GetScopeLatestSyncState(pkv ...interface{}) ([]*models.LatestSyncState, errors.Error) {
+	scope, err := scopeSrv.ModelSrvHelper.FindByPk(pkv...)
+	if err != nil {
+		return nil, err
+	}
+	s := *scope
+	params := plugin.MarshalScopeParams(s.ScopeParams())
+	scopeSrv.log.Debug("scope: %#+v, params: %+v", s, params)
+	rows, err := scopeSrv.db.Cursor(
+		dal.Select("raw_data_table, MAX(latest_success_start) as latest_success_start"),

Review Comment:
   I don't think group by `raw_data_table` and take the max of `latest_success_start` is needed because `raw_data_table`s  are unique when `raw_data_params` is specified.
   I think simply select the needed field would be sufficient.



##########
backend/helpers/srvhelper/scope_service_helper.go:
##########
@@ -93,6 +93,44 @@ func (scopeSrv *ScopeSrvHelper[C, S, SC]) GetScopeDetail(includeBlueprints bool,
 	return scopeDetail, nil
 }
 
+func (scopeSrv *ScopeSrvHelper[C, S, SC]) GetScopeLatestSyncState(pkv ...interface{}) ([]*models.LatestSyncState, errors.Error) {
+	scope, err := scopeSrv.ModelSrvHelper.FindByPk(pkv...)
+	if err != nil {
+		return nil, err
+	}
+	s := *scope
+	params := plugin.MarshalScopeParams(s.ScopeParams())
+	scopeSrv.log.Debug("scope: %#+v, params: %+v", s, params)
+	rows, err := scopeSrv.db.Cursor(
+		dal.Select("raw_data_table, MAX(latest_success_start) as latest_success_start"),
+		dal.From("_devlake_collector_latest_state"),
+		dal.Where("raw_data_params = ?", params),
+		dal.Groupby("raw_data_table"),
+	)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	scopeSyncStates := []*models.LatestSyncState{}

Review Comment:
   Seems like we are returning the whole list anyway, I suggest that we use the `All` method instead of using `Cursor` to make it simpler and readable



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1429598220


##########
backend/plugins/bitbucket/api/scope.go:
##########
@@ -82,6 +82,25 @@ func GetScopeList(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, er
 	return scopeHelper.GetScopeList(input)
 }
 
+// GetScopeDispatcher process BitBucket repos get request
+// @Summary process BitBucket repos get request
+// @Description process BitBucket repos get request
+// @Tags plugins/bitbucket
+// @Param connectionId path int true "connection ID"
+// @Param scopeId path string true "repo ID"
+// @Success 200  {object} ScopeRes
+// @Failure 400  {object} shared.ApiBody "Bad Request"
+// @Failure 500  {object} shared.ApiBody "Internal Error"
+// @Router /plugins/bitbucket/connections/{connectionId}/scopes/{scopeId} [GET]
+func GetScopeDispatcher(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {

Review Comment:
   I see, I think we should chose either one of the following:
   1. encode the `{string_a}/{string_b}` before sending it to the backend, cc @mintsweet 
   2. define a new resource for that, e.g. `/plugins/bitbucket/sync_state?connectionId=&scopeId=&table=`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1429628146


##########
backend/plugins/bitbucket/api/scope.go:
##########
@@ -82,6 +82,25 @@ func GetScopeList(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, er
 	return scopeHelper.GetScopeList(input)
 }
 
+// GetScopeDispatcher process BitBucket repos get request
+// @Summary process BitBucket repos get request
+// @Description process BitBucket repos get request
+// @Tags plugins/bitbucket
+// @Param connectionId path int true "connection ID"
+// @Param scopeId path string true "repo ID"
+// @Success 200  {object} ScopeRes
+// @Failure 400  {object} shared.ApiBody "Bad Request"
+// @Failure 500  {object} shared.ApiBody "Internal Error"
+// @Router /plugins/bitbucket/connections/{connectionId}/scopes/{scopeId} [GET]
+func GetScopeDispatcher(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {

Review Comment:
   all of course.. we must unified them



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1429493392


##########
backend/plugins/bitbucket/api/scope.go:
##########
@@ -82,6 +82,25 @@ func GetScopeList(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, er
 	return scopeHelper.GetScopeList(input)
 }
 
+// GetScopeDispatcher process BitBucket repos get request
+// @Summary process BitBucket repos get request
+// @Description process BitBucket repos get request
+// @Tags plugins/bitbucket
+// @Param connectionId path int true "connection ID"
+// @Param scopeId path string true "repo ID"
+// @Success 200  {object} ScopeRes
+// @Failure 400  {object} shared.ApiBody "Bad Request"
+// @Failure 500  {object} shared.ApiBody "Internal Error"
+// @Router /plugins/bitbucket/connections/{connectionId}/scopes/{scopeId} [GET]
+func GetScopeDispatcher(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {

Review Comment:
   This seems hacky, why not add a new route and we can describe  the response body in another Swagger API correctly?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "d4x1 (via GitHub)" <gi...@apache.org>.
d4x1 commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1429619427


##########
backend/plugins/bitbucket/api/scope.go:
##########
@@ -82,6 +82,25 @@ func GetScopeList(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, er
 	return scopeHelper.GetScopeList(input)
 }
 
+// GetScopeDispatcher process BitBucket repos get request
+// @Summary process BitBucket repos get request
+// @Description process BitBucket repos get request
+// @Tags plugins/bitbucket
+// @Param connectionId path int true "connection ID"
+// @Param scopeId path string true "repo ID"
+// @Success 200  {object} ScopeRes
+// @Failure 400  {object} shared.ApiBody "Bad Request"
+// @Failure 500  {object} shared.ApiBody "Internal Error"
+// @Router /plugins/bitbucket/connections/{connectionId}/scopes/{scopeId} [GET]
+func GetScopeDispatcher(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {

Review Comment:
   @klesh 
   way2: Do you mean we change all `latest-sync-state` to `/plugins/{plugin_name}/sync_state` in all plugins ? or just bitbucket ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "d4x1 (via GitHub)" <gi...@apache.org>.
d4x1 commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1430835453


##########
backend/plugins/jenkins/impl/impl.go:
##########
@@ -189,11 +189,14 @@ func (p Jenkins) ApiResources() map[string]map[string]plugin.ApiResourceHandler
 		// "connections/:connectionId/search-remote-scopes": {
 		// 	"GET": api.SearchRemoteScopes,
 		// },
-		"connections/:connectionId/scopes/*scopeId": {
+		"connections/:connectionId/scopes/:scopeId": {

Review Comment:
   Got it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "abeizn (via GitHub)" <gi...@apache.org>.
abeizn commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1430824410


##########
backend/plugins/jenkins/impl/impl.go:
##########
@@ -189,11 +189,14 @@ func (p Jenkins) ApiResources() map[string]map[string]plugin.ApiResourceHandler
 		// "connections/:connectionId/search-remote-scopes": {
 		// 	"GET": api.SearchRemoteScopes,
 		// },
-		"connections/:connectionId/scopes/*scopeId": {
+		"connections/:connectionId/scopes/:scopeId": {

Review Comment:
   This may also need to be handled like bitbucket



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "d4x1 (via GitHub)" <gi...@apache.org>.
d4x1 commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1429542548


##########
backend/plugins/bitbucket/api/scope.go:
##########
@@ -82,6 +82,25 @@ func GetScopeList(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, er
 	return scopeHelper.GetScopeList(input)
 }
 
+// GetScopeDispatcher process BitBucket repos get request
+// @Summary process BitBucket repos get request
+// @Description process BitBucket repos get request
+// @Tags plugins/bitbucket
+// @Param connectionId path int true "connection ID"
+// @Param scopeId path string true "repo ID"
+// @Success 200  {object} ScopeRes
+// @Failure 400  {object} shared.ApiBody "Bad Request"
+// @Failure 500  {object} shared.ApiBody "Internal Error"
+// @Router /plugins/bitbucket/connections/{connectionId}/scopes/{scopeId} [GET]
+func GetScopeDispatcher(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {

Review Comment:
   Let me explain it in detail.
   `ScopeID` in plugin bitbucket is a string with format `{string_a}/{string_b}`, eg `thenicetgp/ptest`, then the gin path will be `/plugins/bitbucket/connections/{connectionId}/scopes/{string_a}/{string_b}`, but `latest-sync-state` api should be `/plugins/bitbucket/connections/{connectionId}/scopes/{scope_id}/latest-sync-state`. Gin cannot handle it with params in path, more detail: https://github.com/gin-gonic/gin/issues/1446. 
   So we have to handle it specially.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1429493392


##########
backend/plugins/bitbucket/api/scope.go:
##########
@@ -82,6 +82,25 @@ func GetScopeList(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, er
 	return scopeHelper.GetScopeList(input)
 }
 
+// GetScopeDispatcher process BitBucket repos get request
+// @Summary process BitBucket repos get request
+// @Description process BitBucket repos get request
+// @Tags plugins/bitbucket
+// @Param connectionId path int true "connection ID"
+// @Param scopeId path string true "repo ID"
+// @Success 200  {object} ScopeRes
+// @Failure 400  {object} shared.ApiBody "Bad Request"
+// @Failure 500  {object} shared.ApiBody "Internal Error"
+// @Router /plugins/bitbucket/connections/{connectionId}/scopes/{scopeId} [GET]
+func GetScopeDispatcher(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {

Review Comment:
   This seems hacky, why not add a new route and we can Swagger Doc the response body correctly?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feat 6520 [incubator-devlake]

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #6628:
URL: https://github.com/apache/incubator-devlake/pull/6628#discussion_r1429493392


##########
backend/plugins/bitbucket/api/scope.go:
##########
@@ -82,6 +82,25 @@ func GetScopeList(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, er
 	return scopeHelper.GetScopeList(input)
 }
 
+// GetScopeDispatcher process BitBucket repos get request
+// @Summary process BitBucket repos get request
+// @Description process BitBucket repos get request
+// @Tags plugins/bitbucket
+// @Param connectionId path int true "connection ID"
+// @Param scopeId path string true "repo ID"
+// @Success 200  {object} ScopeRes
+// @Failure 400  {object} shared.ApiBody "Bad Request"
+// @Failure 500  {object} shared.ApiBody "Internal Error"
+// @Router /plugins/bitbucket/connections/{connectionId}/scopes/{scopeId} [GET]
+func GetScopeDispatcher(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {

Review Comment:
   This seems hacky, why not add a new route and we can describe the response body correctly?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org