You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/27 13:24:55 UTC
[GitHub] [beam] pawelpasterz opened a new pull request #11534: Push ioit tests metrics to influxdb
pawelpasterz opened a new pull request #11534:
URL: https://github.com/apache/beam/pull/11534
Introduce new InfluxDB publisher for java IT tests.
This PR depends on https://github.com/apache/beam/pull/11491
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pawelpasterz commented on a change in pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
pawelpasterz commented on a change in pull request #11534:
URL: https://github.com/apache/beam/pull/11534#discussion_r419991095
##########
File path: sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
##########
@@ -43,4 +43,22 @@
String getBigQueryTable();
void setBigQueryTable(@Nullable String tableName);
+
+ @Description("InfluxDB measurement to publish results to.")
+ @Nullable
+ String getInfluxMeasurement();
+
+ void setInfluxMeasurement(@Nullable String measurement);
+
+ @Description("InfluxDB host.")
+ @Nullable
+ String getInfluxHost();
+
+ void setInfluxHost(@Nullable String host);
+
+ @Description("InfluxDB databse.")
Review comment:
👍 Thanks for catching this, shame on me
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on a change in pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on a change in pull request #11534:
URL: https://github.com/apache/beam/pull/11534#discussion_r419314057
##########
File path: sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/InfluxDBSettings.java
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testutils.publishing;
+
+import static java.util.Objects.isNull;
+
+import java.util.Arrays;
+import java.util.Objects;
+
+public final class InfluxDBSettings {
Review comment:
You can omit all the bolierplate here (including the builder) by using AutoValue, did you consider it?
##########
File path: sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/IOITMetrics.java
##########
@@ -70,4 +72,21 @@ public static void publish(
}
ConsoleResultPublisher.publish(results, uuid, timestamp);
}
+
+ public void publishToInflux(final InfluxDBSettings settings) {
+ MetricsReader reader = new MetricsReader(result, namespace);
+ Collection<NamedTestResult> namedTestResults = reader.readAll(metricSuppliers);
+
+ InfluxDBPublisher.publishWithSettings(namedTestResults, settings);
+ }
+
+ public static void publishToInflux(
+ final String uuid,
+ final String timestamp,
+ final Collection<NamedTestResult> results,
+ final InfluxDBSettings settings) {
+
+ InfluxDBPublisher.publishWithSettings(results, settings);
+ ConsoleResultPublisher.publish(results, uuid, timestamp);
Review comment:
For the time being this is unnecessary, the results will be published to the console twice, since BQ publisher does the same thing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624011434
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia removed a comment on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia removed a comment on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624022133
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-623964358
Run Java XmlIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624022383
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pawelpasterz commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
pawelpasterz commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-623103333
R: @mwalenia
Hey @mwalenia , please take a look and let me know what do you think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624022133
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia removed a comment on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia removed a comment on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-623949926
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-623964143
Run Java TextIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia removed a comment on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia removed a comment on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624490349
Run Load Tests Java GBK Dataflow Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624008068
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pawelpasterz commented on a change in pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
pawelpasterz commented on a change in pull request #11534:
URL: https://github.com/apache/beam/pull/11534#discussion_r419966077
##########
File path: sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/InfluxDBSettings.java
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testutils.publishing;
+
+import static java.util.Objects.isNull;
+
+import java.util.Arrays;
+import java.util.Objects;
+
+public final class InfluxDBSettings {
Review comment:
Yeah, I was thinking about it in the first place. We have here some custom behavior (like `copyWithMeasurement` or default value for host) which, correct me if I'm wrong, could not be achieved with `AutoValue`.
If it is possible I can rewrite this implementation with AV
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on a change in pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on a change in pull request #11534:
URL: https://github.com/apache/beam/pull/11534#discussion_r419976575
##########
File path: sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
##########
@@ -43,4 +43,22 @@
String getBigQueryTable();
void setBigQueryTable(@Nullable String tableName);
+
+ @Description("InfluxDB measurement to publish results to.")
+ @Nullable
+ String getInfluxMeasurement();
+
+ void setInfluxMeasurement(@Nullable String measurement);
+
+ @Description("InfluxDB host.")
+ @Nullable
+ String getInfluxHost();
+
+ void setInfluxHost(@Nullable String host);
+
+ @Description("InfluxDB databse.")
Review comment:
Nit: typo :)
##########
File path: sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/InfluxDBSettings.java
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testutils.publishing;
+
+import static java.util.Objects.isNull;
+
+import java.util.Arrays;
+import java.util.Objects;
+
+public final class InfluxDBSettings {
Review comment:
I think copying with measurement is possible with AV, but I'm not sure about default values. I think we can leave this as is and refactor later if necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-623969795
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pawelpasterz commented on a change in pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
pawelpasterz commented on a change in pull request #11534:
URL: https://github.com/apache/beam/pull/11534#discussion_r419990601
##########
File path: sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/InfluxDBSettings.java
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testutils.publishing;
+
+import static java.util.Objects.isNull;
+
+import java.util.Arrays;
+import java.util.Objects;
+
+public final class InfluxDBSettings {
Review comment:
I have some ideas how to refactor metrics and publishing in general. My idea was to refactor it once we move to Influx
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624488829
Run Java TextIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624029385
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-623950160
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624488674
run java precommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia removed a comment on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia removed a comment on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624488674
run java precommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624469096
If the tests pass, I'm giving it a LGTM :) Thanks for the contribution!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624488913
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624468886
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-623355311
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624490349
Run Load Tests Java GBK Dataflow Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624488565
Run Java TextIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-623949926
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia commented on pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
mwalenia commented on pull request #11534:
URL: https://github.com/apache/beam/pull/11534#issuecomment-624490466
Run Load Tests Java GBK Dataflow Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pawelpasterz commented on a change in pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
pawelpasterz commented on a change in pull request #11534:
URL: https://github.com/apache/beam/pull/11534#discussion_r419966406
##########
File path: sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/IOITMetrics.java
##########
@@ -70,4 +72,21 @@ public static void publish(
}
ConsoleResultPublisher.publish(results, uuid, timestamp);
}
+
+ public void publishToInflux(final InfluxDBSettings settings) {
+ MetricsReader reader = new MetricsReader(result, namespace);
+ Collection<NamedTestResult> namedTestResults = reader.readAll(metricSuppliers);
+
+ InfluxDBPublisher.publishWithSettings(namedTestResults, settings);
+ }
+
+ public static void publishToInflux(
+ final String uuid,
+ final String timestamp,
+ final Collection<NamedTestResult> results,
+ final InfluxDBSettings settings) {
+
+ InfluxDBPublisher.publishWithSettings(results, settings);
+ ConsoleResultPublisher.publish(results, uuid, timestamp);
Review comment:
👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pawelpasterz commented on a change in pull request #11534: Push ioit tests metrics to influxdb
Posted by GitBox <gi...@apache.org>.
pawelpasterz commented on a change in pull request #11534:
URL: https://github.com/apache/beam/pull/11534#discussion_r419966077
##########
File path: sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/InfluxDBSettings.java
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testutils.publishing;
+
+import static java.util.Objects.isNull;
+
+import java.util.Arrays;
+import java.util.Objects;
+
+public final class InfluxDBSettings {
Review comment:
Yeah, I was thinking about it in the first place. We have some custom behavior here (like `copyWithMeasurement` or default value for host) which, correct me if I'm wrong, could not be achieved with `AutoValue`.
If it is possible I can rewrite this implementation with AV
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org