You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "klesh (via GitHub)" <gi...@apache.org> on 2024/01/26 13:05:34 UTC
[I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
klesh opened a new issue, #6220:
URL: https://github.com/apache/incubator-devlake/issues/6220
## What and why to refactor
Currently, we have 2 types of ApiCollectors
1. `ApiCollector` - Stateless API collection helper
2. `ApiCollectorStateManager` - Stateful API collection helper on top of `ApiCollector` which tracks the `LastSuceededTime`
work in 4 modes
1. `FullSync` from `ApiCollector`: Note that it means `ApiCollector` doesn't offer any help regarding Incremental collection, and the developer must implement the feature on his/her own.
2. `IncrementalSync by updatedAt` from `ApiCollectorStateManager`: offers aids when API supports filtering records by the `updatedAt` field.
3. `IncrementalSync by createdAt` from `ApiCollectorStateManager`: offers aids when API supports filtering records by the `createdAt` or returned records are sorted by `createdAt`. **Useful for collecting Pipelines/Jobs** that can be Finalized(Can not be re-opened) and Automated Short-lived entities (no human operation involved and it will be closed within days)
4. `IncrementalSync by createdAt plus refresh unfinished records` from `ApiCollectorStateManager`: offers refreshing unfished records based on the previous mode. **Useful for collecting PR while the API doesn't support filtering by `updatedAt`**
The **problem**: Developers must figure out how they work, the details/differences of each mode, and which one to use with what parameters.
Take the jenkins builds collector as an example:
```
func CollectApiBuilds(taskCtx plugin.SubTaskContext) errors.Error {
data := taskCtx.GetData().(*JenkinsTaskData)
db := taskCtx.GetDal()
collector, err := helper.NewStatefulApiCollectorForFinalizableEntity(helper.FinalizableApiCollectorArgs{
RawDataSubTaskArgs: helper.RawDataSubTaskArgs{
...
},
ApiClient: data.ApiClient,
CollectNewRecordsByList: helper.FinalizableApiCollectorListArgs{
PageSize: 100,
Concurrency: 10,
FinalizableApiCollectorCommonArgs: helper.FinalizableApiCollectorCommonArgs{
UrlTemplate: fmt.Sprintf("%sjob/%s/api/json", data.Options.JobPath, data.Options.JobName),
Query: func(reqData *helper.RequestData, createdAfter *time.Time) (url.Values, errors.Error) {
...
},
ResponseParser: func(res *http.Response) ([]json.RawMessage, errors.Error) {
...
},
},
GetCreated: func(item json.RawMessage) (time.Time, errors.Error) {
...
},
},
CollectUnfinishedDetails: &helper.FinalizableApiCollectorDetailArgs{
BuildInputIterator: func() (helper.Iterator, errors.Error) {
...
},
FinalizableApiCollectorCommonArgs: helper.FinalizableApiCollectorCommonArgs{
UrlTemplate: fmt.Sprintf("%sjob/%s/{{ .Input.Number }}/api/json?tree=number,url,result,timestamp,id,duration,estimatedDuration,building",
data.Options.JobPath, data.Options.JobName),
ResponseParser: func(res *http.Response) ([]json.RawMessage, errors.Error) {
...
},
},
},
})
if err != nil {
return err
}
return collector.Execute()
}
```
**It is hard to just copy the code and make a new collector correctly.**
1. One would need to understand all collectors and all modes
2. One would need to understand what can API endpoint can offer
## Describe the solution you'd like
The problem can be solved by offering a document with detailed descriptions/tutorials of how to use them against different APIs, but it is a huge effort, for both author and readers.
I believe a better solution is to refactor the ApiCollector and make it Declarative:
```go
collector := &DeclartiveApiCollector{
RawDataSubTaskArgs: ...,
ApiClient: ...,
TimeAfterFiltering: {
ByUpdateAt: {
Supported: true/false,
Via: QUERY_STRING,
KeyName: "updated_at_after"
},
ByCreatedAt: {
RecordFinalizable: true, // panic if false was given
Strategy: COLLECT_FINALIZED_RECORDS_ONLY | REFRESH_UNFINALIZED,
Via: QUERYSTRING | SORTED_RECORDS, // returned records are sorted by `createdAt`
KeyName: "",
GetCreated: func(record json.RawMessage) {...} // extract createdAt from the json
},
},
}
return collector.Execute()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6220:
URL: https://github.com/apache/incubator-devlake/issues/6220#issuecomment-2409462211
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6220:
URL: https://github.com/apache/incubator-devlake/issues/6220#issuecomment-1859342278
This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6220:
URL: https://github.com/apache/incubator-devlake/issues/6220#issuecomment-2048636214
This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6220: [Refactor][helper] Declarative ApiCollector
URL: https://github.com/apache/incubator-devlake/issues/6220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6220:
URL: https://github.com/apache/incubator-devlake/issues/6220#issuecomment-2425319764
This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6220:
URL: https://github.com/apache/incubator-devlake/issues/6220#issuecomment-1849142951
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6220: [Refactor][helper] Declarative ApiCollector
URL: https://github.com/apache/incubator-devlake/issues/6220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6220: [Refactor][helper] Declarative ApiCollector
URL: https://github.com/apache/incubator-devlake/issues/6220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6220:
URL: https://github.com/apache/incubator-devlake/issues/6220#issuecomment-2033314639
This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6220:
URL: https://github.com/apache/incubator-devlake/issues/6220#issuecomment-2159537371
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6220:
URL: https://github.com/apache/incubator-devlake/issues/6220#issuecomment-2282944228
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6220:
URL: https://github.com/apache/incubator-devlake/issues/6220#issuecomment-2021695442
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][helper] Declarative ApiCollector [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6220: [Refactor][helper] Declarative ApiCollector
URL: https://github.com/apache/incubator-devlake/issues/6220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org