You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "keon94 (via GitHub)" <gi...@apache.org> on 2023/05/04 07:12:03 UTC
[GitHub] [incubator-devlake] keon94 opened a new pull request, #5085: test: Manual Integration tests
keon94 opened a new pull request, #5085:
URL: https://github.com/apache/incubator-devlake/pull/5085
### Summary
A directory to contain e2e/integration tests hitting real data sources that won't run as part of automation.
### Does this close any open issues?
Closes n/a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on pull request #5085: test: Manual Integration tests
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on PR #5085:
URL: https://github.com/apache/incubator-devlake/pull/5085#issuecomment-1535425330
@klesh Thanks. That might be a bit hard to do because each plugin will have its own datamodels for Scopes and TxRules (and to some extent Connections) with their own unique fields. I'll try to add a few more test suites for other plugins at the meantime, and see if there's any room for consolidating logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on pull request #5085: test: Manual Integration tests
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on PR #5085:
URL: https://github.com/apache/incubator-devlake/pull/5085#issuecomment-1577542313
Currently broken until all of the following issues/PRs are closed:
- #5345
- #5368
- #5369
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on pull request #5085: test: Manual Integration tests
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on PR #5085:
URL: https://github.com/apache/incubator-devlake/pull/5085#issuecomment-1587714425
> This is a great addition to tests. Like Klesh, I'd like to see how to get a generic version of it. It looks like the plugin-specific connection and scopeConfig can be replaced with `map[string]any` here? Also it would be great to be able to pass test config with tokens as secrets from CI.
@CamilleTeruel @klesh
I'd like to avoid using `map[string]any` in favor of using actual structs - this way if the models change the tests will break at compile time and we can easily detect that. Also, it's easier for readbility sake.
But even with that aside, these tests interact with specific model fields that are specific to each plugin (e.g. The specific Scopes and ScopeConfig), and generifying that won't be simple at all and will make the tests more complex. I personally think the current level of abstraction is sufficient for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #5085: test: Manual Integration tests
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #5085:
URL: https://github.com/apache/incubator-devlake/pull/5085#discussion_r1226931415
##########
backend/test/e2e/manual/pagerduty/pagerduty_test.go:
##########
@@ -0,0 +1,164 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package pagerduty
+
+import (
+ "fmt"
+ "github.com/apache/incubator-devlake/core/config"
+ "github.com/apache/incubator-devlake/core/models"
+ "github.com/apache/incubator-devlake/core/plugin"
+ "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
+ "github.com/apache/incubator-devlake/plugins/pagerduty/impl"
+ pluginmodels "github.com/apache/incubator-devlake/plugins/pagerduty/models"
+ "github.com/apache/incubator-devlake/test/helper"
+ "github.com/stretchr/testify/require"
+ "testing"
+ "time"
+)
+
+const pluginName = "pagerduty"
+
+func TestPagerDutyPlugin(t *testing.T) {
+ cfg := helper.GetTestConfig[TestConfig]()
+ client := helper.ConnectLocalServer(t, &helper.LocalClientConfig{
+ ServerPort: 8089,
+ DbURL: config.GetConfig().GetString("E2E_DB_URL"),
+ CreateServer: true,
+ DropDb: false,
+ TruncateDb: true,
+ Plugins: map[string]plugin.PluginMeta{
+ pluginName: &impl.PagerDuty{},
+ },
+ })
+ client.SetTimeout(0)
+ client.SetPipelineTimeout(0)
+ connection := createConnection(cfg, client)
+ t.Run("blueprint v200", func(t *testing.T) {
+ serviceScopes := helper.RemoteScopesOutput{}
+ var scopeData []any
+ for {
+ serviceScopes = client.RemoteScopes(helper.RemoteScopesQuery{
+ PluginName: pluginName,
+ ConnectionId: connection.ID,
+ PageToken: serviceScopes.NextPageToken,
+ Params: nil,
+ })
+ for _, remoteScope := range serviceScopes.Children {
+ if remoteScope.Type == "scope" {
+ data := helper.Cast[pluginmodels.Service](remoteScope.Data)
+ for _, serviceName := range cfg.Services {
+ if serviceName == data.Name {
+ scopeData = append(scopeData, &data)
+ }
+ }
+ }
+ }
+ if serviceScopes.NextPageToken == "" {
+ break
+ }
+ }
+ createdScopes := helper.Cast[[]*pluginmodels.Service](client.CreateScope(pluginName, connection.ID, scopeData...))
+ require.True(t, len(createdScopes) > 0)
+ uniqueString := time.Now().Format(time.RFC3339)
+ outputProject := createProject(client, fmt.Sprintf("pagerduty-project-%s-%s", pluginName, uniqueString))
+ var bpScopes []*plugin.BlueprintScopeV200
+ for _, scope := range createdScopes {
+ bpScopes = append(bpScopes, &plugin.BlueprintScopeV200{
+ Id: scope.Id,
+ Name: fmt.Sprintf("pagerduty-blueprint-v200-%s", uniqueString),
+ })
+ }
+ bp := client.CreateBasicBlueprintV2(connection.Name, &helper.BlueprintV2Config{
+ Connection: &plugin.BlueprintConnectionV200{
+ Plugin: pluginName,
+ ConnectionId: connection.ID,
+ Scopes: bpScopes,
+ },
+ TimeAfter: cfg.TimeSince,
+ SkipOnFail: false,
+ ProjectName: outputProject.Name,
+ })
+ // get the project ... should have a reference to the blueprint now
+ outputProject = client.GetProject(outputProject.Name)
+ require.Equal(t, bp.Name, outputProject.Blueprint.Name)
+ fmt.Printf("=========================Triggering blueprint for project %s =========================\n", outputProject.Name)
+ client.TriggerBlueprint(bp.ID)
+ createdScopesList := client.ListScopes(pluginName, connection.ID, true)
+ require.True(t, len(createdScopesList) > 0)
+ for _, scope := range createdScopesList {
+ scopeCast := helper.Cast[pluginmodels.Service](scope.Scope)
+ fmt.Printf("Deleting scope %s\n", scopeCast.Id)
+ client.DeleteScope(pluginName, connection.ID, scopeCast.Id, false)
+ fmt.Printf("Deleted scope %s\n", scopeCast.Id)
+ }
+ })
+ fmt.Println("======DONE======")
+ time.Sleep(120 * time.Minute)
Review Comment:
Yeah this was intentional so I could have time to check the DB tables and validate their data. There's technically no need to poll here because the Trigger function already does this. We can just remove these Sleeps and do this instead:
```go
pipeline := client.TriggerBlueprint(bp.ID)
require.Equal(t, models.TASK_COMPLETED, pipeline.Status)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on a diff in pull request #5085: test: Manual Integration tests
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on code in PR #5085:
URL: https://github.com/apache/incubator-devlake/pull/5085#discussion_r1227141807
##########
backend/test/e2e/manual/pagerduty/pagerduty_test.go:
##########
@@ -0,0 +1,164 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package pagerduty
+
+import (
+ "fmt"
+ "github.com/apache/incubator-devlake/core/config"
+ "github.com/apache/incubator-devlake/core/models"
+ "github.com/apache/incubator-devlake/core/plugin"
+ "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
+ "github.com/apache/incubator-devlake/plugins/pagerduty/impl"
+ pluginmodels "github.com/apache/incubator-devlake/plugins/pagerduty/models"
+ "github.com/apache/incubator-devlake/test/helper"
+ "github.com/stretchr/testify/require"
+ "testing"
+ "time"
+)
+
+const pluginName = "pagerduty"
+
+func TestPagerDutyPlugin(t *testing.T) {
+ cfg := helper.GetTestConfig[TestConfig]()
+ client := helper.ConnectLocalServer(t, &helper.LocalClientConfig{
+ ServerPort: 8089,
+ DbURL: config.GetConfig().GetString("E2E_DB_URL"),
+ CreateServer: true,
+ DropDb: false,
+ TruncateDb: true,
+ Plugins: map[string]plugin.PluginMeta{
+ pluginName: &impl.PagerDuty{},
+ },
+ })
+ client.SetTimeout(0)
+ client.SetPipelineTimeout(0)
+ connection := createConnection(cfg, client)
+ t.Run("blueprint v200", func(t *testing.T) {
+ serviceScopes := helper.RemoteScopesOutput{}
+ var scopeData []any
+ for {
+ serviceScopes = client.RemoteScopes(helper.RemoteScopesQuery{
+ PluginName: pluginName,
+ ConnectionId: connection.ID,
+ PageToken: serviceScopes.NextPageToken,
+ Params: nil,
+ })
+ for _, remoteScope := range serviceScopes.Children {
+ if remoteScope.Type == "scope" {
+ data := helper.Cast[pluginmodels.Service](remoteScope.Data)
+ for _, serviceName := range cfg.Services {
+ if serviceName == data.Name {
+ scopeData = append(scopeData, &data)
+ }
+ }
+ }
+ }
+ if serviceScopes.NextPageToken == "" {
+ break
+ }
+ }
+ createdScopes := helper.Cast[[]*pluginmodels.Service](client.CreateScope(pluginName, connection.ID, scopeData...))
+ require.True(t, len(createdScopes) > 0)
+ uniqueString := time.Now().Format(time.RFC3339)
+ outputProject := createProject(client, fmt.Sprintf("pagerduty-project-%s-%s", pluginName, uniqueString))
+ var bpScopes []*plugin.BlueprintScopeV200
+ for _, scope := range createdScopes {
+ bpScopes = append(bpScopes, &plugin.BlueprintScopeV200{
+ Id: scope.Id,
+ Name: fmt.Sprintf("pagerduty-blueprint-v200-%s", uniqueString),
+ })
+ }
+ bp := client.CreateBasicBlueprintV2(connection.Name, &helper.BlueprintV2Config{
+ Connection: &plugin.BlueprintConnectionV200{
+ Plugin: pluginName,
+ ConnectionId: connection.ID,
+ Scopes: bpScopes,
+ },
+ TimeAfter: cfg.TimeSince,
+ SkipOnFail: false,
+ ProjectName: outputProject.Name,
+ })
+ // get the project ... should have a reference to the blueprint now
+ outputProject = client.GetProject(outputProject.Name)
+ require.Equal(t, bp.Name, outputProject.Blueprint.Name)
+ fmt.Printf("=========================Triggering blueprint for project %s =========================\n", outputProject.Name)
+ client.TriggerBlueprint(bp.ID)
+ createdScopesList := client.ListScopes(pluginName, connection.ID, true)
+ require.True(t, len(createdScopesList) > 0)
+ for _, scope := range createdScopesList {
+ scopeCast := helper.Cast[pluginmodels.Service](scope.Scope)
+ fmt.Printf("Deleting scope %s\n", scopeCast.Id)
+ client.DeleteScope(pluginName, connection.ID, scopeCast.Id, false)
+ fmt.Printf("Deleted scope %s\n", scopeCast.Id)
+ }
+ })
+ fmt.Println("======DONE======")
+ time.Sleep(120 * time.Minute)
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on pull request #5085: test: Manual Integration tests
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on PR #5085:
URL: https://github.com/apache/incubator-devlake/pull/5085#issuecomment-1584112937
@klesh Could you give this a test on your machine using the Readme I've created? I want to gauge the convenience of using this for others. You can try running the Gitlab test. I've tested it and works for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on pull request #5085: test: Manual Integration tests
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on PR #5085:
URL: https://github.com/apache/incubator-devlake/pull/5085#issuecomment-1534246758
The idea is solid, I think we could take it further like offering some kind of `testhelper` for the scenario since all plugins share the same behavior pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 merged pull request #5085: test: Manual Integration tests
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 merged PR #5085:
URL: https://github.com/apache/incubator-devlake/pull/5085
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] CamilleTeruel commented on a diff in pull request #5085: test: Manual Integration tests
Posted by "CamilleTeruel (via GitHub)" <gi...@apache.org>.
CamilleTeruel commented on code in PR #5085:
URL: https://github.com/apache/incubator-devlake/pull/5085#discussion_r1226868835
##########
backend/test/e2e/manual/pagerduty/pagerduty_test.go:
##########
@@ -0,0 +1,164 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package pagerduty
+
+import (
+ "fmt"
+ "github.com/apache/incubator-devlake/core/config"
+ "github.com/apache/incubator-devlake/core/models"
+ "github.com/apache/incubator-devlake/core/plugin"
+ "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
+ "github.com/apache/incubator-devlake/plugins/pagerduty/impl"
+ pluginmodels "github.com/apache/incubator-devlake/plugins/pagerduty/models"
+ "github.com/apache/incubator-devlake/test/helper"
+ "github.com/stretchr/testify/require"
+ "testing"
+ "time"
+)
+
+const pluginName = "pagerduty"
+
+func TestPagerDutyPlugin(t *testing.T) {
+ cfg := helper.GetTestConfig[TestConfig]()
+ client := helper.ConnectLocalServer(t, &helper.LocalClientConfig{
+ ServerPort: 8089,
+ DbURL: config.GetConfig().GetString("E2E_DB_URL"),
+ CreateServer: true,
+ DropDb: false,
+ TruncateDb: true,
+ Plugins: map[string]plugin.PluginMeta{
+ pluginName: &impl.PagerDuty{},
+ },
+ })
+ client.SetTimeout(0)
+ client.SetPipelineTimeout(0)
+ connection := createConnection(cfg, client)
+ t.Run("blueprint v200", func(t *testing.T) {
+ serviceScopes := helper.RemoteScopesOutput{}
+ var scopeData []any
+ for {
+ serviceScopes = client.RemoteScopes(helper.RemoteScopesQuery{
+ PluginName: pluginName,
+ ConnectionId: connection.ID,
+ PageToken: serviceScopes.NextPageToken,
+ Params: nil,
+ })
+ for _, remoteScope := range serviceScopes.Children {
+ if remoteScope.Type == "scope" {
+ data := helper.Cast[pluginmodels.Service](remoteScope.Data)
+ for _, serviceName := range cfg.Services {
+ if serviceName == data.Name {
+ scopeData = append(scopeData, &data)
+ }
+ }
+ }
+ }
+ if serviceScopes.NextPageToken == "" {
+ break
+ }
+ }
+ createdScopes := helper.Cast[[]*pluginmodels.Service](client.CreateScope(pluginName, connection.ID, scopeData...))
+ require.True(t, len(createdScopes) > 0)
+ uniqueString := time.Now().Format(time.RFC3339)
+ outputProject := createProject(client, fmt.Sprintf("pagerduty-project-%s-%s", pluginName, uniqueString))
+ var bpScopes []*plugin.BlueprintScopeV200
+ for _, scope := range createdScopes {
+ bpScopes = append(bpScopes, &plugin.BlueprintScopeV200{
+ Id: scope.Id,
+ Name: fmt.Sprintf("pagerduty-blueprint-v200-%s", uniqueString),
+ })
+ }
+ bp := client.CreateBasicBlueprintV2(connection.Name, &helper.BlueprintV2Config{
+ Connection: &plugin.BlueprintConnectionV200{
+ Plugin: pluginName,
+ ConnectionId: connection.ID,
+ Scopes: bpScopes,
+ },
+ TimeAfter: cfg.TimeSince,
+ SkipOnFail: false,
+ ProjectName: outputProject.Name,
+ })
+ // get the project ... should have a reference to the blueprint now
+ outputProject = client.GetProject(outputProject.Name)
+ require.Equal(t, bp.Name, outputProject.Blueprint.Name)
+ fmt.Printf("=========================Triggering blueprint for project %s =========================\n", outputProject.Name)
+ client.TriggerBlueprint(bp.ID)
+ createdScopesList := client.ListScopes(pluginName, connection.ID, true)
+ require.True(t, len(createdScopesList) > 0)
+ for _, scope := range createdScopesList {
+ scopeCast := helper.Cast[pluginmodels.Service](scope.Scope)
+ fmt.Printf("Deleting scope %s\n", scopeCast.Id)
+ client.DeleteScope(pluginName, connection.ID, scopeCast.Id, false)
+ fmt.Printf("Deleted scope %s\n", scopeCast.Id)
+ }
+ })
+ fmt.Println("======DONE======")
+ time.Sleep(120 * time.Minute)
Review Comment:
This makes the test hang even after blueprint is finished.
Can we poll instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org