You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/07/19 16:57:14 UTC
[GitHub] [beam] roger-mike opened a new pull request, #22344: [BEAM-13004] DebeziumIO Load Test
roger-mike opened a new pull request, #22344:
URL: https://github.com/apache/beam/pull/22344
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1220894798
Run Spotless PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] andoni-guzman commented on a diff in pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
andoni-guzman commented on code in PR #22344:
URL: https://github.com/apache/beam/pull/22344#discussion_r963047620
##########
.test-infra/kubernetes/postgres/postgres-service-for-debezium.yml:
##########
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: postgres-for-dev
+ labels:
+ name: postgres
+spec:
+ ports:
+ - port: 5432
+ selector:
+ name: postgres
+ type: LoadBalancer
+
+---
+
+apiVersion: v1
+kind: ReplicationController
+metadata:
+ name: postgres
+spec:
+ replicas: 1
+ selector:
+ name: postgres
+ template:
+ metadata:
+ name: postgres
+ labels:
+ name: postgres
+ spec:
+ containers:
+ - name: postgres
+ image: postgres
+ args: ["-c", "wal_level=logical","-c","max_wal_senders=4","-c","max_replication_slots=4"]
Review Comment:
Hi! We need to use this args to enable Postgres Change Data Capture (CDC).
The reason for having in different files is try to avoid errors with another tests that use postgres-for-dev file and have different files for postgres with CDC disable and enable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1279296385
> @roger-mike sorry I missed the last seed job comment. Do you still need one run? What is the next action on this PR?
Yes, could you run it again, please? Just want to check that the fixes work đź‘Ť
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1213513658
Yep, I can help here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1251210728
It is running now - https://ci-beam.apache.org/job/beam_SeedJob/10303/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1194406541
Run Python Debezium Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1199915395
I'm not immediately sure - I don't have a _ton_ of area expertise here, it certainly looks like a dependency issue though.
Unfortunately, I'm going on vacation in <1 hour, so I probably won't be able to work through this one with you - @TheNeuralBit would you mind helping out here with any seed jobs that need to be run (and maybe some ideas on what might be happening)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1220894681
Run RAT PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1213218637
> If we're to the point where you're iterating on Python code and not the job definition, we should probably find another way to iterate, either:
>
> * Can you run the test locally? Maybe we need to get you access to some GCP resources? or
> * We could go ahead and merge the new job with it failing, then you can iterate yourself
I can't run the test locally due to some dependencies issues, right now I think we can merge the new job definition and iterate on the python code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1258460159
Running now - https://ci-beam.apache.org/job/beam_SeedJob/10334/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1271513630
Reminder, please take a look at this pr: @AnandInguva
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #22344:
URL: https://github.com/apache/beam/pull/22344#discussion_r931313667
##########
.test-infra/jenkins/job_PerformanceTests_Debezium.groovy:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as common
+import Kubernetes
+import InfluxDBCredentialsHelper
+
+String jobName = "beam_PerformanceTests_Debezium"
+
+// String kubernetesYmlPath = "src/.test-infra/kubernetes/postgres/\n" +
+// "postgres-service-for-debezium.yml"
+
+
+String task = ":sdks:python:apache_beam:testing:load_tests:run -PloadTest.mainClass=apache_beam.testing.load_tests.debezium_performance -Prunner=DirectRunner"
+
+static String parseOptions(Map<String, ?> options) {
+options.collect { entry ->
+ // Flags are indicated by null values
+ if (entry.value == null) {
+ "--${entry.key}"
+ } else {
+ "--${entry.key}=$entry.value".replace('\"', '\\\"').replace('\'', '\\\'')
+ }
+}.join(' ')
+}
+
+job(jobName) {
+ common.setTopLevelMainJobProperties(delegate)
+ common.setAutoJob(delegate, 'H H/12 * * *')
+ common.enablePhraseTriggeringFromPullRequest(
+ delegate,
+ 'Python Debezium Performance Test',
+ 'Run Python Debezium Performance Test')
+ InfluxDBCredentialsHelper.useCredentials(delegate)
+
+// String namespace = common.getKubernetesNamespace(jobName)
+// String kubeconfig = common.getKubeconfigLocationForNamespace(namespace)
+ String postgresHostName = "LOAD_BALANCER_IP"
+
+// Kubernetes k8s = Kubernetes.create(delegate, kubeconfig, namespace)
+// k8s.apply(common.makePathAbsolute(kubernetesYmlPath))
+// k8s.loadBalancerIP("postgres-for-dev", postgresHostName)
+
+ //(0..2).each { k8s.loadBalancerIP("outside-$it", "Postgres_IP_$it") }
+
+ Map pipelineOptions = [
+ //kubernetes_host : "\$LOAD_BALANCER_IP",
+ kubernetes_host : '146.148.98.7',
+ kubernetes_port : "5432",
+ postgres_user: 'postgres',
+ postgres_password: 'uuinkks',
+ input_options : '\'{' +
+ '"num_records": 20000000 }\''
+ ]
+
+
+ steps {
+ gradle {
+ rootBuildScriptDir(common.checkoutDir)
+ dependsOn("sdks:java:io:debezium:expansion-service:shadowJar")
Review Comment:
The seed job failed, it doesn't like this line - https://ci-beam.apache.org/job/beam_SeedJob/10045/console
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1204514235
Run Seed Job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1209603828
Another failure:
```
Processing DSL script .test-infra/jenkins/job_PerformanceTests_Debezium.groovy
ERROR: (job_PerformanceTests_Debezium.groovy, line 63) No such property: LoadTestBuilder for class: javaposse.jobdsl.dsl.helpers.step.GradleContext
Sending e-mails to: builds@beam.apache.org
Finished: FAILURE
```
https://ci-beam.apache.org/job/beam_SeedJob/10112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1209696260
@TheNeuralBit There was an error when running the Kubernetes part, maybe because the path had a `\n`. Could you run it again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1251222268
Run Python Debezium Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1258474515
Run Python Debezium Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1194407671
I reran the seed job and started https://ci-beam.apache.org/job/beam_PerformanceTests_Debezium/1/. The seed job will run automatically again soon overwriting this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1199921840
Sure I can help here. Unfortunately I'm also going to have limited availability this afternoon, but I'll be around next week.
In this case this looks like a situation where we the test depends on a Java dependency, we need to add an annotation for that in the gradle file. Let me see if I can find an example of that..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1258428312
Hi @damccorm could you run another seed job? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1279413102
Run Python Debezium Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1213477726
> I can't run the test locally due to some dependencies issues, right now I think we can merge the new job definition and iterate on the python code.
Ok, if you'd like to go that route please get the existing checks (PythonFormatter, PythonLint, Spotless, ...) passing first. If you'd like we could also help debug the dependency issues.
@damccorm - could you take this back? I will be on vacation for the next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #22344:
URL: https://github.com/apache/beam/pull/22344#discussion_r951825957
##########
.test-infra/kubernetes/postgres/postgres-service-for-debezium.yml:
##########
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: postgres-for-dev
+ labels:
+ name: postgres
+spec:
+ ports:
+ - port: 5432
+ selector:
+ name: postgres
+ type: LoadBalancer
+
+---
+
+apiVersion: v1
+kind: ReplicationController
+metadata:
+ name: postgres
+spec:
+ replicas: 1
+ selector:
+ name: postgres
+ template:
+ metadata:
+ name: postgres
+ labels:
+ name: postgres
+ spec:
+ containers:
+ - name: postgres
+ image: postgres
+ args: ["-c", "wal_level=logical","-c","max_wal_senders=4","-c","max_replication_slots=4"]
+
+
+ env:
Review Comment:
Nit - weird spacing here
##########
.test-infra/kubernetes/postgres/postgres-service-for-debezium.yml:
##########
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: postgres-for-dev
Review Comment:
```suggestion
name: postgres-for-debezium
```
##########
.test-infra/jenkins/job_PerformanceTests_Debezium.groovy:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as common
+import Kubernetes
+import InfluxDBCredentialsHelper
+import LoadTestsBuilder
+
+String jobName = "beam_PerformanceTests_Debezium"
+
+String kubernetesYmlPath = "src/.test-infra/kubernetes/postgres/postgres-service-for-debezium.yml"
+
+
+String task = ":sdks:python:apache_beam:testing:load_tests:run -PloadTest.mainClass=apache_beam.testing.load_tests.debezium_performance_test -Prunner=DirectRunner"
Review Comment:
Do we really want to be running this on the direct runner? That seems unusual?
##########
sdks/python/apache_beam/testing/load_tests/debezium_performance_test.py:
##########
@@ -0,0 +1,170 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Performance test for debezium.
+
+The purpose of this test is verify that Python's connector ReadFromDebezium
+work propertly, for this, the test create a postgresql database inside a
+kubernetes pod and stream querys inside of the database for 20 minutes.
+After that ReadFromDebezium checks if everything goes well
+
+Example test run:
+
+python -m apache_beam.testing.load_tests.debezium_performance
+
+or:
+
+./gradlew
+ -PloadTest.mainClass=apache_beam.testing.load_tests.debezium_performance \
+-Prunner=DirectRunner :sdks:python:apache_beam:testing:load_tests:run
+"""
+
+import logging
+import random
+import time
+from concurrent.futures import ThreadPoolExecutor
+
+import psycopg2
+
+from apache_beam.io.debezium import DriverClassName
+from apache_beam.io.debezium import ReadFromDebezium
+from apache_beam.testing.load_tests.load_test import LoadTest
+
+
+class DebeziumLoadTest(LoadTest):
+ def __init__(self):
+ super().__init__()
+ self.kubernetes_host = self.pipeline.get_option('kubernetes_host')
+ self.kubernetes_port = self.pipeline.get_option('kubernetes_port')
+ self.postgres_user = self.pipeline.get_option('postgres_user')
+ self.postgres_password = self.pipeline.get_option('postgres_password')
+
+ self.username = self.postgres_user
+ self.password = self.postgres_password
+
+ self.database = 'postgres'
+ self.port = self.kubernetes_port
+ self.host = self.kubernetes_host
+ self.connector_class = DriverClassName.POSTGRESQL
+ self.connection_properties = [
+ "database.dbname=postgres",
+ "database.server.name=postgres",
+ "database.include.list=postgres",
+ "include.schema.changes=false",
+ "plugin.name=pgoutput"
+ ]
+
+ def initConnection(self):
+ connection = psycopg2.connect(
+ host=self.host,
+ database=self.database,
+ user=self.username,
+ password=self.password)
+ return connection
+
+ def randomInsertTest(self):
+ connection = self.initConnection()
+ insert = 0
+ cursor = connection.cursor()
+ createTable = """
+ CREATE TABLE IF NOT EXISTS postgres(
+ id NUMERIC,
+ word VARCHAR(50),
+ number NUMERIC,
+ date DATE,
+ bool BOOLEAN
+ )
+ """
+ cursor.execute(createTable)
+ alterTableReplica = "ALTER TABLE postgres REPLICA IDENTITY FULL;"
+ cursor.execute(alterTableReplica)
+ startTime = time.time()
+ testDuration = 60 * 22
+ timeFlag = True
+
+ logging.debug('INSERTING RANDOMLY')
+ while timeFlag:
+
+ action = random.randint(1, 10)
+ if action == 1: # Delete
Review Comment:
What is the purpose of inserting randomness here? Won't that make it harder for us to get a deterministic signal about whether performance has actually degraded?
##########
.test-infra/kubernetes/postgres/postgres-service-for-debezium.yml:
##########
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: postgres-for-dev
+ labels:
+ name: postgres
+spec:
+ ports:
+ - port: 5432
+ selector:
+ name: postgres
+ type: LoadBalancer
+
+---
+
+apiVersion: v1
+kind: ReplicationController
+metadata:
+ name: postgres
+spec:
+ replicas: 1
+ selector:
+ name: postgres
+ template:
+ metadata:
+ name: postgres
+ labels:
+ name: postgres
+ spec:
+ containers:
+ - name: postgres
+ image: postgres
+ args: ["-c", "wal_level=logical","-c","max_wal_senders=4","-c","max_replication_slots=4"]
Review Comment:
Why do we need these args? Seems like that is the only difference between this and postgres-for-dev, are we able to just reuse that yml file instead?
##########
.test-infra/jenkins/job_PerformanceTests_Debezium.groovy:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as common
+import Kubernetes
+import InfluxDBCredentialsHelper
+import LoadTestsBuilder
+
+String jobName = "beam_PerformanceTests_Debezium"
+
+String kubernetesYmlPath = "src/.test-infra/kubernetes/postgres/postgres-service-for-debezium.yml"
+
+
+String task = ":sdks:python:apache_beam:testing:load_tests:run -PloadTest.mainClass=apache_beam.testing.load_tests.debezium_performance_test -Prunner=DirectRunner"
Review Comment:
Also, any reason to pull this out into a variable? We only use it once and it seems like the convention is to inline it. Same question for the kubernetesYmlPath.
If we do keep it, we should fix the spacing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1194335070
@roger-mike Your seed job might have overwritten by the periodically running seed job(it runs every 6 hours)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1259830691
@damccorm can you run a seed job, please? Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1279412932
Just ran it - https://ci-beam.apache.org/job/beam_SeedJob/10426/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Lizzfox commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by "Lizzfox (via GitHub)" <gi...@apache.org>.
Lizzfox commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1448294932
Run Python Debezium Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1582512829
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] closed pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #22344: [BEAM-13004] DebeziumIO Load Test
URL: https://github.com/apache/beam/pull/22344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1191676671
@roger-mike the seed job is failing because of an issue that has been fixed in the main repo (#22223) could you please update your branch with the latest changes from master?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1194327227
Hi @damccorm, the job appears as disabled, could you enable it and run the seed job again? Thank you.
https://ci-beam.apache.org/job/beam_PerformanceTests_Debezium/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1192726466
Done - https://ci-beam.apache.org/job/beam_SeedJob/10017/console
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1231688620
stop reviewer notifications
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1212398854
If we're to the point where you're iterating on Python code and not the job definition, we should probably find another way to iterate, either:
- Can you run the test locally? Maybe we need to get you access to some GCP resources? or
- We could go ahead and merge the new job with it failing, then you can iterate yourself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1592971975
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1199923013
Here is one example, although it doesn't translate directly. When making the XVR (Cross-language validates runner) suites we add a dependency on the necessary expansion service: https://github.com/apache/beam/blob/e5e3cb25ca4fc2e31c10eb3dbda8289c6bfc7140/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L2292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1204566083
Sorry I forgot I had to run the job manually for non-committers. It's finished now: https://ci-beam.apache.org/job/beam_SeedJob/10079/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1204417761
Hi @TheNeuralBit could you run the seed job again? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1209654984
Run Python Debezium Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1209655242
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1191662221
Run Seed Job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1222706607
@damccorm Could you take a look? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #22344:
URL: https://github.com/apache/beam/pull/22344#discussion_r963615778
##########
sdks/python/apache_beam/testing/load_tests/debezium_performance_test.py:
##########
@@ -0,0 +1,170 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Performance test for debezium.
+
+The purpose of this test is verify that Python's connector ReadFromDebezium
+work propertly, for this, the test create a postgresql database inside a
+kubernetes pod and stream querys inside of the database for 20 minutes.
+After that ReadFromDebezium checks if everything goes well
+
+Example test run:
+
+python -m apache_beam.testing.load_tests.debezium_performance
+
+or:
+
+./gradlew
+ -PloadTest.mainClass=apache_beam.testing.load_tests.debezium_performance \
+-Prunner=DirectRunner :sdks:python:apache_beam:testing:load_tests:run
+"""
+
+import logging
+import random
+import time
+from concurrent.futures import ThreadPoolExecutor
+
+import psycopg2
+
+from apache_beam.io.debezium import DriverClassName
+from apache_beam.io.debezium import ReadFromDebezium
+from apache_beam.testing.load_tests.load_test import LoadTest
+
+
+class DebeziumLoadTest(LoadTest):
+ def __init__(self):
+ super().__init__()
+ self.kubernetes_host = self.pipeline.get_option('kubernetes_host')
+ self.kubernetes_port = self.pipeline.get_option('kubernetes_port')
+ self.postgres_user = self.pipeline.get_option('postgres_user')
+ self.postgres_password = self.pipeline.get_option('postgres_password')
+
+ self.username = self.postgres_user
+ self.password = self.postgres_password
+
+ self.database = 'postgres'
+ self.port = self.kubernetes_port
+ self.host = self.kubernetes_host
+ self.connector_class = DriverClassName.POSTGRESQL
+ self.connection_properties = [
+ "database.dbname=postgres",
+ "database.server.name=postgres",
+ "database.include.list=postgres",
+ "include.schema.changes=false",
+ "plugin.name=pgoutput"
+ ]
+
+ def initConnection(self):
+ connection = psycopg2.connect(
+ host=self.host,
+ database=self.database,
+ user=self.username,
+ password=self.password)
+ return connection
+
+ def randomInsertTest(self):
+ connection = self.initConnection()
+ insert = 0
+ cursor = connection.cursor()
+ createTable = """
+ CREATE TABLE IF NOT EXISTS postgres(
+ id NUMERIC,
+ word VARCHAR(50),
+ number NUMERIC,
+ date DATE,
+ bool BOOLEAN
+ )
+ """
+ cursor.execute(createTable)
+ alterTableReplica = "ALTER TABLE postgres REPLICA IDENTITY FULL;"
+ cursor.execute(alterTableReplica)
+ startTime = time.time()
+ testDuration = 60 * 22
+ timeFlag = True
+
+ logging.debug('INSERTING RANDOMLY')
+ while timeFlag:
+
+ action = random.randint(1, 10)
+ if action == 1: # Delete
Review Comment:
Yeah, I think using fixed numbers of each type is a better approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1192711109
@damccorm done đź‘Ť , could you run the seed job again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1196950887
Hi @damccorm, could you run the seed job again? Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1251154528
@damccorm could you run a seed job? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1206884471
Hi @TheNeuralBit could you run the seed job again? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1208505794
Seed job failed with:
```
ERROR: startup failed:
job_PerformanceTests_Debezium.groovy: 22: unable to resolve class LoadTestBuilder
@ line 22, column 1.
import LoadTestBuilder
```
https://ci-beam.apache.org/job/beam_SeedJob/10108/console
I saw that there's also https://github.com/apache/beam/pull/22135 for this task, do we need both?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1189354599
# [Codecov](https://codecov.io/gh/apache/beam/pull/22344?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#22344](https://codecov.io/gh/apache/beam/pull/22344?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ad0e18) into [master](https://codecov.io/gh/apache/beam/commit/316c969b9c373177986ee0fbc794ed2887244328?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (316c969) will **decrease** coverage by `0.06%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #22344 +/- ##
==========================================
- Coverage 74.01% 73.94% -0.07%
==========================================
Files 702 704 +2
Lines 92845 93249 +404
==========================================
+ Hits 68716 68954 +238
- Misses 22864 23030 +166
Partials 1265 1265
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.43% <0.00%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/22344?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...he\_beam/testing/load\_tests/debezium\_performance.py](https://codecov.io/gh/apache/beam/pull/22344/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9sb2FkX3Rlc3RzL2RlYmV6aXVtX3BlcmZvcm1hbmNlLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/internal/dill\_pickler.py](https://codecov.io/gh/apache/beam/pull/22344/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvZGlsbF9waWNrbGVyLnB5) | `85.61% <0.00%> (-1.44%)` | :arrow_down: |
| [...ython/apache\_beam/io/gcp/bigquery\_read\_internal.py](https://codecov.io/gh/apache/beam/pull/22344/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3JlYWRfaW50ZXJuYWwucHk=) | `53.36% <0.00%> (-1.05%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/22344/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.12% <0.00%> (-0.62%)` | :arrow_down: |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/22344/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `94.34% <0.00%> (-0.62%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/22344/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.59% <0.00%> (-0.34%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/22344/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.32% <0.00%> (-0.24%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/22344/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.54% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/external.py](https://codecov.io/gh/apache/beam/pull/22344/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbC5weQ==) | `79.73% <0.00%> (-0.11%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/filebasedsink.py](https://codecov.io/gh/apache/beam/pull/22344/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWJhc2Vkc2luay5weQ==) | `95.90% <0.00%> (-0.08%)` | :arrow_down: |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/22344/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1189475119
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @AnandInguva for label python.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1209484008
> Seed job failed with:
>
> ```
> ERROR: startup failed:
> job_PerformanceTests_Debezium.groovy: 22: unable to resolve class LoadTestBuilder
> @ line 22, column 1.
> import LoadTestBuilder
> ```
>
> https://ci-beam.apache.org/job/beam_SeedJob/10108/console
>
> I saw that there's also #22135 for this task, do we need both?
Sorry, just a typo. We'll close #22135, is not needed anymore. Could you run the seed job again? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1279247827
@roger-mike sorry I missed the last seed job comment. Do you still need one run? What is the next action on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1204566231
Run Python Debezium Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1209630499
> Another failure:
>
> ```
> Processing DSL script .test-infra/jenkins/job_PerformanceTests_Debezium.groovy
> ERROR: (job_PerformanceTests_Debezium.groovy, line 63) No such property: LoadTestBuilder for class: javaposse.jobdsl.dsl.helpers.step.GradleContext
> Sending e-mails to: builds@beam.apache.org
> Finished: FAILURE
> ```
>
> https://ci-beam.apache.org/job/beam_SeedJob/10112
Fixed, can you run it again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1231584661
Reminder, please take a look at this pr: @AnandInguva
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] andoni-guzman commented on a diff in pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
andoni-guzman commented on code in PR #22344:
URL: https://github.com/apache/beam/pull/22344#discussion_r963052171
##########
sdks/python/apache_beam/testing/load_tests/debezium_performance_test.py:
##########
@@ -0,0 +1,170 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Performance test for debezium.
+
+The purpose of this test is verify that Python's connector ReadFromDebezium
+work propertly, for this, the test create a postgresql database inside a
+kubernetes pod and stream querys inside of the database for 20 minutes.
+After that ReadFromDebezium checks if everything goes well
+
+Example test run:
+
+python -m apache_beam.testing.load_tests.debezium_performance
+
+or:
+
+./gradlew
+ -PloadTest.mainClass=apache_beam.testing.load_tests.debezium_performance \
+-Prunner=DirectRunner :sdks:python:apache_beam:testing:load_tests:run
+"""
+
+import logging
+import random
+import time
+from concurrent.futures import ThreadPoolExecutor
+
+import psycopg2
+
+from apache_beam.io.debezium import DriverClassName
+from apache_beam.io.debezium import ReadFromDebezium
+from apache_beam.testing.load_tests.load_test import LoadTest
+
+
+class DebeziumLoadTest(LoadTest):
+ def __init__(self):
+ super().__init__()
+ self.kubernetes_host = self.pipeline.get_option('kubernetes_host')
+ self.kubernetes_port = self.pipeline.get_option('kubernetes_port')
+ self.postgres_user = self.pipeline.get_option('postgres_user')
+ self.postgres_password = self.pipeline.get_option('postgres_password')
+
+ self.username = self.postgres_user
+ self.password = self.postgres_password
+
+ self.database = 'postgres'
+ self.port = self.kubernetes_port
+ self.host = self.kubernetes_host
+ self.connector_class = DriverClassName.POSTGRESQL
+ self.connection_properties = [
+ "database.dbname=postgres",
+ "database.server.name=postgres",
+ "database.include.list=postgres",
+ "include.schema.changes=false",
+ "plugin.name=pgoutput"
+ ]
+
+ def initConnection(self):
+ connection = psycopg2.connect(
+ host=self.host,
+ database=self.database,
+ user=self.username,
+ password=self.password)
+ return connection
+
+ def randomInsertTest(self):
+ connection = self.initConnection()
+ insert = 0
+ cursor = connection.cursor()
+ createTable = """
+ CREATE TABLE IF NOT EXISTS postgres(
+ id NUMERIC,
+ word VARCHAR(50),
+ number NUMERIC,
+ date DATE,
+ bool BOOLEAN
+ )
+ """
+ cursor.execute(createTable)
+ alterTableReplica = "ALTER TABLE postgres REPLICA IDENTITY FULL;"
+ cursor.execute(alterTableReplica)
+ startTime = time.time()
+ testDuration = 60 * 22
+ timeFlag = True
+
+ logging.debug('INSERTING RANDOMLY')
+ while timeFlag:
+
+ action = random.randint(1, 10)
+ if action == 1: # Delete
Review Comment:
Hi, one of the request of the test was having a continuous random data streaming ( inserst, updates and deletes). This is the reason of having this randomness in the test.
I agree that this approximation is not the best to get a good performance metric and could be a good change modify the behavior to get a more deterministic test.
Maybe having a fixed number of inserts, updates and deletes could be a good way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #22344:
URL: https://github.com/apache/beam/pull/22344#discussion_r963617372
##########
.test-infra/kubernetes/postgres/postgres-service-for-debezium.yml:
##########
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: postgres-for-dev
+ labels:
+ name: postgres
+spec:
+ ports:
+ - port: 5432
+ selector:
+ name: postgres
+ type: LoadBalancer
+
+---
+
+apiVersion: v1
+kind: ReplicationController
+metadata:
+ name: postgres
+spec:
+ replicas: 1
+ selector:
+ name: postgres
+ template:
+ metadata:
+ name: postgres
+ labels:
+ name: postgres
+ spec:
+ containers:
+ - name: postgres
+ image: postgres
+ args: ["-c", "wal_level=logical","-c","max_wal_senders=4","-c","max_replication_slots=4"]
Review Comment:
> The reason for having in different files is try to avoid errors with another tests that use postgres-for-dev file and have different files for postgres with CDC disable and enable.
Is there a reason we can't just turn on CDC for all tests? I'm not too worried about changes to this file breaking tests since we should catch that with our CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1258481767
Run Python Debezium Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1259652028
Run Python PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1258467894
Run Python Debezium Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1372171145
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1194421460
Run Python Debezium Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1199902961
Hi, @damccorm any idea why this error happens when running the job? https://ci-beam.apache.org/job/beam_PerformanceTests_Debezium/2/console
```
13:02:05 Traceback (most recent call last):
13:02:05 File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
13:02:05 return _run_code(code, main_globals, None,
13:02:05 File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
13:02:05 exec(code, run_globals)
13:02:05 File "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Debezium/src/sdks/python/apache_beam/testing/load_tests/debezium_performance.py", line 190, in <module>
13:02:05 debeziumTest.createPipeline()
13:02:05 File "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Debezium/src/sdks/python/apache_beam/testing/load_tests/debezium_performance.py", line 156, in createPipeline
13:02:05 pipeline | 'Read from debezium' >> ReadFromDebezium(
13:02:05 File "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Debezium/src/sdks/python/apache_beam/io/debezium.py", line 168, in __init__
13:02:05 self.expansion_service = expansion_service or default_io_expansion_service()
13:02:05 File "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Debezium/src/sdks/python/apache_beam/io/debezium.py", line 98, in default_io_expansion_service
13:02:05 return BeamJarExpansionService(
13:02:05 File "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Debezium/src/sdks/python/apache_beam/transforms/external.py", line 820, in __init__
13:02:05 path_to_jar = subprocess_server.JavaJarServer.path_to_beam_jar(
13:02:05 File "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Debezium/src/sdks/python/apache_beam/utils/subprocess_server.py", line 243, in path_to_beam_jar
13:02:05 raise RuntimeError(
13:02:05 RuntimeError: /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Debezium/src/sdks/java/io/debezium/expansion-service/build/libs/beam-sdks-java-io-debezium-expansion-service-2.41.0-SNAPSHOT.jar not found. Please build the server with
13:02:05 cd /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Debezium/src; ./gradlew sdks:java:io:debezium:expansion-service:shadowJar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] roger-mike commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
roger-mike commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1220894580
Run PythonDocs PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #22344: [BEAM-13004] DebeziumIO Load Test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #22344:
URL: https://github.com/apache/beam/pull/22344#issuecomment-1212397676
Run Python Debezium Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org