You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "keon94 (via GitHub)" <gi...@apache.org> on 2023/04/12 06:51:32 UTC
[GitHub] [incubator-devlake] keon94 opened a new pull request, #4906: [feat-4762]: Support deleting data by scope
keon94 opened a new pull request, #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906
### Summary
What does this PR do?
### Does this close any open issues?
Closes #4762
### Screenshots
Include any relevant screenshots here.
### Other Information
Any other information that is important to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#discussion_r1169206116
##########
backend/core/models/domainlayer/domaininfo/domaininfo.go:
##########
@@ -47,7 +47,7 @@ func GetDomainTablesInfo() []Tabler {
&code.RefsPrCherrypick{},
&code.Repo{},
&code.RepoCommit{},
- &code.RepoLanguage{},
Review Comment:
commented out for now because the table for it doesnt exist yet (until I find out why)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#discussion_r1180937468
##########
backend/python/pydevlake/pydevlake/stream.py:
##########
@@ -80,9 +80,14 @@ class StreamRawModel(RawModel, table=True):
__tablename__ = table_name
self._raw_model = StreamRawModel
- RawModel.metadata.create_all(session.get_bind())
+ if session is not None:
Review Comment:
I've removed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#discussion_r1179616357
##########
backend/test/e2e/remote/python_plugin_test.go:
##########
@@ -159,4 +159,8 @@ func TestBlueprintV200(t *testing.T) {
project := client.GetProject(projectName)
require.Equal(t, blueprint.Name, project.Blueprint.Name)
client.TriggerBlueprint(blueprint.ID)
+
+ bps := client.DeleteScope(PLUGIN_NAME, connection.ID, scope.Id)
+ require.Equal(t, 1, len(bps))
+ require.Equal(t, blueprint.ID, bps[0].ID)
Review Comment:
This is more of an e2e for a full blueprint lifecycle - triggering is just a part of it, and it includes creating a scope. Also, need the blueprint to be triggered in advance anyway so that we have plugin data in the tables (which gets deleted when DeleteScope is called)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] CamilleTeruel commented on a diff in pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "CamilleTeruel (via GitHub)" <gi...@apache.org>.
CamilleTeruel commented on code in PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#discussion_r1179283670
##########
backend/test/e2e/remote/python_plugin_test.go:
##########
@@ -159,4 +159,8 @@ func TestBlueprintV200(t *testing.T) {
project := client.GetProject(projectName)
require.Equal(t, blueprint.Name, project.Blueprint.Name)
client.TriggerBlueprint(blueprint.ID)
+
+ bps := client.DeleteScope(PLUGIN_NAME, connection.ID, scope.Id)
+ require.Equal(t, 1, len(bps))
+ require.Equal(t, blueprint.ID, bps[0].ID)
Review Comment:
This test is about triggering a blueprint. Scope deletion should have a dedicated test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#discussion_r1169207079
##########
backend/plugins/dora/impl/impl.go:
##########
@@ -121,7 +121,7 @@ func (p Dora) MakeMetricPluginPipelinePlanV200(projectName string, options json.
stageDeploymentCommitdiff := plugin.PipelineStage{
{
Plugin: "refdiff",
- Subtasks: []string{"calculateProjectDeploymentCommitsDiff"},
+ Subtasks: []string{"calculateDeploymentCommitsDiff"},
Review Comment:
temp fix for dora plugin failure for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] CamilleTeruel commented on a diff in pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "CamilleTeruel (via GitHub)" <gi...@apache.org>.
CamilleTeruel commented on code in PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#discussion_r1179278317
##########
backend/python/pydevlake/pydevlake/stream.py:
##########
@@ -80,9 +80,14 @@ class StreamRawModel(RawModel, table=True):
__tablename__ = table_name
self._raw_model = StreamRawModel
- RawModel.metadata.create_all(session.get_bind())
+ if session is not None:
Review Comment:
When is the session None?
##########
backend/test/e2e/remote/python_plugin_test.go:
##########
@@ -159,4 +159,8 @@ func TestBlueprintV200(t *testing.T) {
project := client.GetProject(projectName)
require.Equal(t, blueprint.Name, project.Blueprint.Name)
client.TriggerBlueprint(blueprint.ID)
+
+ bps := client.DeleteScope(PLUGIN_NAME, connection.ID, scope.Id)
+ require.Equal(t, 1, len(bps))
+ require.Equal(t, blueprint.ID, bps[0].ID)
Review Comment:
This test is about triggering a blueprint. Scope deletion should have its own dedicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on a diff in pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#discussion_r1170719813
##########
backend/helpers/pluginhelper/api/batch_save_divider.go:
##########
@@ -31,13 +31,14 @@ import (
// BatchSaveDivider creates and caches BatchSave, this is helpful when dealing with massive amount of data records
// with arbitrary types.
type BatchSaveDivider struct {
- basicRes context.BasicRes
- log log.Logger
- db dal.Dal
- batches map[reflect.Type]*BatchSave
- batchSize int
- table string
- params string
+ basicRes context.BasicRes
+ log log.Logger
+ db dal.Dal
+ batches map[reflect.Type]*BatchSave
+ batchSize int
+ table string
+ params string
+ ignoreOrigin bool
Review Comment:
Why do we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#issuecomment-1532436786
Sample GET request to get all the scopes with the blueprints:
`GET http://localhost:8089/plugins/pagerduty/connections/25/scopes?blueprints=true`
```json
[
{
"NoPKModel": {
"CreatedAt": {},
"RawDataOrigin": {
"RawDataId": 142,
"RawDataParams": "{\"ConnectionId\":25,\"ScopeId\":\"PIKL83L\"}",
"RawDataRemark": "",
"RawDataTable": "_raw_pagerduty_incidents"
},
"UpdatedAt": {}
},
"blueprints": [
{
"name": "pagerduty-conn",
"projectName": "pagerduty-project-pagerduty-2023-05-02T23:08:14-05:00-0",
"mode": "NORMAL",
"plan": [
[
{
"plugin": "org",
"subtasks": [
"setProjectMapping"
],
"options": {
"projectMappings": [
{
"projectName": "pagerduty-project-pagerduty-2023-05-02T23:08:14-05:00-0",
"scopes": [
{
"table": "boards",
"rowId": "pagerduty:Service:25:PIKL83L"
}
]
}
]
}
}
],
[
{
"plugin": "pagerduty",
"subtasks": [
"collectIncidents",
"extractIncidents",
"convertIncidents"
],
"options": {
"ConnectionId": 25,
"PagerdutyTransformationRule": null,
"ServiceId": "PIKL83L",
"ServiceName": "DevService",
"Tasks": null,
"TimeAfter": "2022-09-01T15:04:05Z"
}
}
],
[
{
"plugin": "dora",
"subtasks": [
"generateDeploymentCommits",
"enrichPrevSuccessDeploymentCommits"
],
"options": {
"projectName": "pagerduty-project-pagerduty-2023-05-02T23:08:14-05:00-0"
}
}
],
[
{
"plugin": "refdiff",
"subtasks": [
"calculateDeploymentCommitsDiff"
],
"options": {
"projectName": "pagerduty-project-pagerduty-2023-05-02T23:08:14-05:00-0"
}
}
],
[
{
"plugin": "dora",
"subtasks": [
"calculateChangeLeadTime",
"ConnectIncidentToDeployment"
],
"options": {
"projectName": "pagerduty-project-pagerduty-2023-05-02T23:08:14-05:00-0"
}
}
]
],
"enable": true,
"cronConfig": "manual",
"isManual": true,
"skipOnFail": true,
"labels": [
"test-label"
],
"settings": {
"version": "2.0.0",
"timeAfter": "2022-09-01T15:04:05Z",
"connections": [
{
"plugin": "pagerduty",
"connectionId": 25,
"scopes": [
{
"id": "PIKL83L",
"name": "pagerduty-blueprint-v200-2023-05-02T23:08:14-05:00",
"entities": [
"TICKET"
]
}
]
}
],
"before_plan": null,
"after_plan": null
},
"id": 148,
"createdAt": "2023-05-02T23:08:14.082-05:00",
"updatedAt": "2023-05-02T23:08:14.082-05:00"
}
],
"connectionId": 25,
"id": "PIKL83L",
"name": "DevService",
"url": "https://keon-test.pagerduty.com/service-directory/PIKL83L"
},
{
"NoPKModel": {
"CreatedAt": {},
"RawDataOrigin": {
"RawDataId": 145,
"RawDataParams": "{\"ConnectionId\":25,\"ScopeId\":\"PIKL83L\"}",
"RawDataRemark": "",
"RawDataTable": "_raw_pagerduty_incidents"
},
"UpdatedAt": {}
},
"connectionId": 25,
"id": "POE5KX2",
"name": "Keon-IT",
"url": ""
}
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#discussion_r1170748620
##########
backend/helpers/pluginhelper/api/batch_save_divider.go:
##########
@@ -31,13 +31,14 @@ import (
// BatchSaveDivider creates and caches BatchSave, this is helpful when dealing with massive amount of data records
// with arbitrary types.
type BatchSaveDivider struct {
- basicRes context.BasicRes
- log log.Logger
- db dal.Dal
- batches map[reflect.Type]*BatchSave
- batchSize int
- table string
- params string
+ basicRes context.BasicRes
+ log log.Logger
+ db dal.Dal
+ batches map[reflect.Type]*BatchSave
+ batchSize int
+ table string
+ params string
+ ignoreOrigin bool
Review Comment:
I was using this to do batch save for generic tables, but I guess we don't need it for now anymore. You think this can be useful later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#issuecomment-1532439969
Sample Delete request to delete a scope (data only - don't touch the scopes):
`DELETE "/plugins/pagerduty/connections/26/scopes/PIKL83L?delete_data_only=true"`
Returns nothing.
Sample Delete request to delete a scope (data and the scope). Returns all the blueprints that used this scope.
``DELETE "/plugins/pagerduty/connections/26/scopes/PIKL83L?delete_data_only=false"`` (may omit delete_data_only - defaults to false)
```json
[
{
"name": "pagerduty-conn",
"projectName": "pagerduty-project-pagerduty-2023-05-02T23:18:19-05:00-0",
"mode": "NORMAL",
"plan": [
[
{
"plugin": "org",
"subtasks": [
"setProjectMapping"
],
"options": {
"projectMappings": [
{
"projectName": "pagerduty-project-pagerduty-2023-05-02T23:18:19-05:00-0",
"scopes": [
{
"table": "boards",
"rowId": "pagerduty:Service:27:PIKL83L"
}
]
}
]
}
}
],
[
{
"plugin": "pagerduty",
"subtasks": [
"collectIncidents",
"extractIncidents",
"convertIncidents"
],
"options": {
"ConnectionId": 27,
"PagerdutyTransformationRule": null,
"ServiceId": "PIKL83L",
"ServiceName": "DevService",
"Tasks": null,
"TimeAfter": "2022-09-01T15:04:05Z"
}
}
],
[
{
"plugin": "dora",
"subtasks": [
"generateDeploymentCommits",
"enrichPrevSuccessDeploymentCommits"
],
"options": {
"projectName": "pagerduty-project-pagerduty-2023-05-02T23:18:19-05:00-0"
}
}
],
[
{
"plugin": "refdiff",
"subtasks": [
"calculateDeploymentCommitsDiff"
],
"options": {
"projectName": "pagerduty-project-pagerduty-2023-05-02T23:18:19-05:00-0"
}
}
],
[
{
"plugin": "dora",
"subtasks": [
"calculateChangeLeadTime",
"ConnectIncidentToDeployment"
],
"options": {
"projectName": "pagerduty-project-pagerduty-2023-05-02T23:18:19-05:00-0"
}
}
]
],
"enable": true,
"cronConfig": "manual",
"isManual": true,
"skipOnFail": true,
"labels": [
"test-label"
],
"settings": {
"version": "2.0.0",
"timeAfter": "2022-09-01T15:04:05Z",
"connections": [
{
"plugin": "pagerduty",
"connectionId": 27,
"scopes": null
}
],
"before_plan": null,
"after_plan": null
},
"id": 150,
"createdAt": "2023-05-02T23:18:19.22-05:00",
"updatedAt": "2023-05-02T23:18:21.378-05:00"
}
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 closed pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 closed pull request #4906: [feat-4762]: Support deleting data by scope
URL: https://github.com/apache/incubator-devlake/pull/4906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#discussion_r1173375600
##########
backend/core/models/domainlayer/domaininfo/domaininfo.go:
##########
@@ -47,7 +47,7 @@ func GetDomainTablesInfo() []Tabler {
&code.RefsPrCherrypick{},
&code.Repo{},
&code.RepoCommit{},
- &code.RepoLanguage{},
Review Comment:
not a feature yet. Will leave it commented as it'll otherwise break the "delete from" logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#discussion_r1185534759
##########
backend/helpers/pluginhelper/api/api_collector.go:
##########
@@ -356,7 +356,7 @@ func (collector *ApiCollector) generateUrl(pager *Pager, input interface{}) (str
var buf bytes.Buffer
err := collector.urlTemplate.Execute(&buf, &RequestData{
Pager: pager,
- Params: collector.args.Params,
+ Params: collector.args.Options.GetParams(),
Review Comment:
The "API Params" of subtasks is now the responsibility of the plugin's options to provide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#issuecomment-1542826361
This PR will be broken down to multiple parts due to reviewers' request, so I am going to close this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on a diff in pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#discussion_r1173718937
##########
backend/helpers/pluginhelper/api/batch_save_divider.go:
##########
@@ -31,13 +31,14 @@ import (
// BatchSaveDivider creates and caches BatchSave, this is helpful when dealing with massive amount of data records
// with arbitrary types.
type BatchSaveDivider struct {
- basicRes context.BasicRes
- log log.Logger
- db dal.Dal
- batches map[reflect.Type]*BatchSave
- batchSize int
- table string
- params string
+ basicRes context.BasicRes
+ log log.Logger
+ db dal.Dal
+ batches map[reflect.Type]*BatchSave
+ batchSize int
+ table string
+ params string
+ ignoreOrigin bool
Review Comment:
Not at the moment, let's delete it for now. We can add it when we actually need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on pull request #4906: [feat-4762]: Support deleting data by scope
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on PR #4906:
URL: https://github.com/apache/incubator-devlake/pull/4906#issuecomment-1532441091
@hezyin @klesh @Startrekzky @yumengwang03
Please see the above Sample API interactions and let me know your thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org