You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/01/20 21:50:04 UTC
[GitHub] [beam] brucearctor opened a new pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
brucearctor opened a new pull request #16578:
URL: https://github.com/apache/beam/pull/16578
Allow Clustering without Partition in BigQuery
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r806265266
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
It looks like this is needed -- from the 'test' ( by removing setting this manually ) the IT failed, but it succeeds if it is in there.
Am inclined to mark this as resolved, but @chamikaramj do advise.
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
It looks like this is needed -- from the 'test' ( by removing setting this manually ) the IT failed, but it succeeds if it is in there. https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/930/testReport/junit/org.apache.beam.sdk.io.gcp.bigquery/BigQueryClusteringIT/testE2EBigQueryClusteringNoPartitionDynamicDestinations/ ...
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
It looks like this is needed -- from the 'test' ( by removing setting this manually ) the IT failed, but it succeeds if these lines setting coder manually is in there. https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/930/testReport/junit/org.apache.beam.sdk.io.gcp.bigquery/BigQueryClusteringIT/testE2EBigQueryClusteringNoPartitionDynamicDestinations/ ...
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
It looks like this is needed -- from the 'test' ( by removing setting this manually ) the IT failed, but it succeeds if these lines (103-106) setting coder manually are in there. https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/930/testReport/junit/org.apache.beam.sdk.io.gcp.bigquery/BigQueryClusteringIT/testE2EBigQueryClusteringNoPartitionDynamicDestinations/ ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040807686
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039358187
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r806265266
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
It looks like this is needed -- from the 'test' ( by removing setting this manually ) the IT failed, but it succeeds if it is in there.
Am inclined to mark this as resolved, but @chamikaramj do advise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 6255ff0. Consider uploading reports for the commit 6255ff0 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...6255ff0](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039480854
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036667346
BTW, also please make sure that this works for all BQ write methods: https://github.com/apache/beam/blob/7fa5387ffac4f2801077f2e55aa2eba7a47036d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1038289276
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036464983
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036858687
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1035865337
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r804343590
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
+ }
+
+ @Override
+ public TableDestination getDestination(ValueInSingleWindow<TableRow> element) {
+ return new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName), null, null, CLUSTERING);
+ }
+
+ @Override
+ public TableDestination getTable(TableDestination destination) {
+ return destination;
+ }
+
+ @Override
+ public TableSchema getSchema(TableDestination destination) {
+ return SCHEMA;
+ }
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringNoPartitionTableFunction() throws Exception {
+ String tableName = "weather_stations_clustered_table_function_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(
+ (ValueInSingleWindow<TableRow> vsw) ->
+ new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName),
+ null,
+ null,
+ CLUSTERING))
+ .withClustering(CLUSTERING)
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(CLUSTERING, table.getClustering());
Review comment:
Sure ... I had set a default table expiration of 5 days in the "ClusteringIT" dataset, to not unnecessarily store IT data. Currently that is 5 days -- do we think we should have a longer retention period?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036462186
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1049203018
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1018159446
@TheNeuralBit -- can you have a look to ensure I'm not way off base with what I'm doing here? I made a new PR here. I think this addresses the current issue as written. Clustering does not appear to work by itself (without TimePartitioning) in all cases, but this does work for the cases that were added. Overall an improvement, but I should I probably add some docs/writeup somewhere for these cases that do work ( to acknowledge is just a subset ). Naturally, will squash all these commits :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039524246
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040521011
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040776501
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head e4c9154. Consider uploading reports for the commit e4c9154 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...e4c9154](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040913248
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039772138
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 8a94697. Consider uploading reports for the commit 8a94697 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...8a94697](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036676686
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036692294
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036461339
run java postcommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036461923
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036703426
You can just try running the same pipeline with three write methods (no need to add additional tests). For testing streaming, for Dataflow, just add the "--streaming" flag. The pipeline will end since the input is bounded but it still runs using the streaming runner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048299304
and, I think I found the correct 'withMethod' for more tests in Jenkins!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 94af35c. Consider uploading reports for the commit 94af35c to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...94af35c](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1049212788
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048278423
If it's hard to run the Jenkins integration test for all write methods (I don't have a command to do so unfortunately) you can just try running a simple pipeline that write to BQ (for example [1]) to make sure the feature works for various write methods.
If it makes easier you can just operate on top of a Maven project created using the archetype given at [2].
If you want to push code from the Beam Gradle repo to local Maven repo you can use the command at [3].
[1] https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryTornadoes.java
[2] https://beam.apache.org/get-started/quickstart-java/
[3] https://cwiki.apache.org/confluence/display/BEAM/Gradle+Tips#GradleTips-PublishamodulesnapshotreleasetomavenLocal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1034384670
Great -- @chamikaramj do take a look if you're up for it or please advise if I should pull in someone else ( I am a committer, and probably being overly cautious, I can also see about pulling anyone else in if you're busy ) -- even though lots of commits, I think the changes are rather straightforward [ once squashed ].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1035861809
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036680033
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036461923
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 39c059f. Consider uploading reports for the commit 39c059f to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...39c059f](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039756885
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039357587
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040908898
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 88f76db. Consider uploading reports for the commit 88f76db to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...88f76db](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 81dbb74. Consider uploading reports for the commit 81dbb74 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...81dbb74](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r806265266
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
It looks like this is needed -- from the 'test' ( by removing setting this manually ) the IT failed, but it succeeds if these lines setting coder manually is in there. https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/930/testReport/junit/org.apache.beam.sdk.io.gcp.bigquery/BigQueryClusteringIT/testE2EBigQueryClusteringNoPartitionDynamicDestinations/ ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1050561219
@chamikaramj -- how's this code look? This is an improvement, that doesn't break any existing tests -- so, I think is OK to merge.
Clustering is not created properly for STORAGE_API_AT_LEAST_ONCE or STORAGE_WRITE_API -- so those will be new tickets that I add. Again, since this is an improvement over existing functionality ( and does not break things ), it seems OK to merge. I might revisit to get those working later, but for now the tests are for 'DEFAULT', and 'FILE_LOADS'.
I will write some follow up tickets for:
a) document this functionality. at least in https://beam.apache.org/documentation/io/built-in/google-bigquery/ , but perhaps anywhere else suggested
b) get clustering working with STORAGE_API_AT_LEAST_ONCE
c) get clustering working with STORAGE_WRITE_API
...? Anything else top of mind ( what else would we want to see, but needn't be captured by this current improvement?
Also, would be curious to learn the implementation differences between STORAGE_API_AT_LEAST_ONCE and STORAGE_WRITE_API ... I wonder if solved for one, if that fixes both.
I see some related issues that I'd like to see addressed ahead of implementing any 'fix' or additional integration tests just for clustering, when using the storage write api --> ex:
https://issues.apache.org/jira/browse/BEAM-13747
and
https://issues.apache.org/jira/browse/BEAM-13753 / https://issues.apache.org/jira/browse/BEAM-13990
If you agree is fine, do squash/merge, or give me the LGTM. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1050420732
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1050269323
Run Spotless PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r804365936
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
+ }
+
+ @Override
+ public TableDestination getDestination(ValueInSingleWindow<TableRow> element) {
+ return new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName), null, null, CLUSTERING);
+ }
+
+ @Override
+ public TableDestination getTable(TableDestination destination) {
+ return destination;
+ }
+
+ @Override
+ public TableSchema getSchema(TableDestination destination) {
+ return SCHEMA;
+ }
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringNoPartitionTableFunction() throws Exception {
+ String tableName = "weather_stations_clustered_table_function_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(
+ (ValueInSingleWindow<TableRow> vsw) ->
+ new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName),
+ null,
+ null,
+ CLUSTERING))
+ .withClustering(CLUSTERING)
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(CLUSTERING, table.getClustering());
Review comment:
Yeah, I mean to check the properties of the data (for example, rowcount, a checksum of the data) in the same IT using an assert. There's an example here: https://github.com/apache/beam/blob/419c03c71d47bc1a4001735e260fcfe083da164d/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java#L70
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1017982233
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039295499
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039501425
Run Java_Examples_Dataflow PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036469392
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036858687
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1037322119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 7c55fc2. Consider uploading reports for the commit 7c55fc2 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...7c55fc2](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036665385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 43bc3c2. Consider uploading reports for the commit 43bc3c2 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...43bc3c2](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head c86590f. Consider uploading reports for the commit c86590f to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...c86590f](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1049009289
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048485722
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head bae4bd8. Consider uploading reports for the commit bae4bd8 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...bae4bd8](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1018015877
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1017983480
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head db2164c. Consider uploading reports for the commit db2164c to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...db2164c](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039756885
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039834544
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1030504992
@kerrydc @chamikaramj - This looks like a BQ change. Could you suggest a potential reviewer for this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1035867172
run java postcommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036676686
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **decrease** coverage by `27.72%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head e587c0a. Consider uploading reports for the commit e587c0a to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
===========================================
- Coverage 74.61% 46.88% -27.73%
===========================================
Files 653 203 -450
Lines 81802 20073 -61729
===========================================
- Hits 61038 9412 -51626
+ Misses 19783 9661 -10122
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...ks/go/pkg/beam/runners/dataflow/dataflowlib/job.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2pvYi5nbw==) | `14.76% <0.00%> (-1.31%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/typex/class.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3R5cGV4L2NsYXNzLmdv) | `82.72% <0.00%> (-0.61%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/exec/fn.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbi5nbw==) | `70.27% <0.00%> (-0.45%)` | :arrow_down: |
| [sdks/go/pkg/beam/runners/dataflow/dataflow.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93Lmdv) | `5.21% <0.00%> (-0.05%)` | :arrow_down: |
| [sdks/go/pkg/beam/runners/direct/direct.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9kaXJlY3QuZ28=) | `1.79% <0.00%> (-0.02%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/exec/hash.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9oYXNoLmdv) | `34.95% <0.00%> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/xlangx/expand.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuZC5nbw==) | `0.00% <0.00%> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/clone.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L2Nsb25lLmdv) | `22.44% <0.00%> (ø)` | |
| ... and [467 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...e587c0a](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1035865337
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1037492898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1038453529
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head f9f706e. Consider uploading reports for the commit f9f706e to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...f9f706e](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040774780
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040718829
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040774780
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1034299582
I'd feel better if @chamikaramj reviewed this since he has more context on the original reasoning for disallowing clustering without partitioning. My involvement here was just filing BEAM-10652 after it came up in a customer issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r804354312
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
+ }
+
+ @Override
+ public TableDestination getDestination(ValueInSingleWindow<TableRow> element) {
+ return new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName), null, null, CLUSTERING);
+ }
+
+ @Override
+ public TableDestination getTable(TableDestination destination) {
+ return destination;
+ }
+
+ @Override
+ public TableSchema getSchema(TableDestination destination) {
+ return SCHEMA;
+ }
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringNoPartitionTableFunction() throws Exception {
+ String tableName = "weather_stations_clustered_table_function_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(
+ (ValueInSingleWindow<TableRow> vsw) ->
+ new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName),
+ null,
+ null,
+ CLUSTERING))
+ .withClustering(CLUSTERING)
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(CLUSTERING, table.getClustering());
Review comment:
The data is being written/created (verified manually). Do you have a suggestion for an Assert to use ( check row count,?check a specific value? ). Clustering *should* only effect the table being created and not the data itself, but doesn't hurt to add some additional checks/tests/assertions.
Perhaps there is an example from another IT that you think would be good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r804343590
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
+ }
+
+ @Override
+ public TableDestination getDestination(ValueInSingleWindow<TableRow> element) {
+ return new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName), null, null, CLUSTERING);
+ }
+
+ @Override
+ public TableDestination getTable(TableDestination destination) {
+ return destination;
+ }
+
+ @Override
+ public TableSchema getSchema(TableDestination destination) {
+ return SCHEMA;
+ }
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringNoPartitionTableFunction() throws Exception {
+ String tableName = "weather_stations_clustered_table_function_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(
+ (ValueInSingleWindow<TableRow> vsw) ->
+ new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName),
+ null,
+ null,
+ CLUSTERING))
+ .withClustering(CLUSTERING)
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(CLUSTERING, table.getClustering());
Review comment:
Sure ... I had set a default table expiration of 5 days in the "BigQueryClusteringIT" dataset, to not unnecessarily store IT data. Currently that is 5 days -- do we think we should have a longer retention period?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 8a30468. Consider uploading reports for the commit 8a30468 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...8a30468](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040778864
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 66786c7. Consider uploading reports for the commit 66786c7 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...66786c7](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1037693888
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036706019
@chamikaramj -- any idea why `Run Java PreCommit` and others do not seem to trigger those testing suites? Is this generally flaky, or am I not doing something that would otherwise be helpful?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head e2ff347. Consider uploading reports for the commit e2ff347 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...e2ff347](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 9a3f81b. Consider uploading reports for the commit 9a3f81b to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...9a3f81b](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040747774
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1018030897
Follow up from https://github.com/apache/beam/pull/15548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head a82157c. Consider uploading reports for the commit a82157c to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...a82157c](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048445380
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048998697
Java Run PreComit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head d6710a0. Consider uploading reports for the commit d6710a0 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...d6710a0](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 3ba86f4. Consider uploading reports for the commit 3ba86f4 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...3ba86f4](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj merged pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
chamikaramj merged pull request #16578:
URL: https://github.com/apache/beam/pull/16578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036665385
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039760694
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039523572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 39c059f. Consider uploading reports for the commit 39c059f to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.65% +0.03%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61474 +436
- Misses 19783 19873 +90
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `96.72% <0.00%> (-1.64%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...39c059f](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039183266
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039183785
Run Spotless PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039792758
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040749946
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040890442
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048290443
Thanks, Cham. Even if hard, doesn't mean not worth doing. **My general philosophy is that if it is important to test once before merging then should be part of ongoing tests....**, so will at least try that.
Though, it doesn't seem to make sense to test the STORAGE_API_AT_LEAST_ONCE , since the test might fail ( if configured to expect exactly-once ), even though the API would be working as 'expected'/advertised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 5b08ef8. Consider uploading reports for the commit 5b08ef8 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...5b08ef8](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048491343
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 2b3c0cf. Consider uploading reports for the commit 2b3c0cf to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...2b3c0cf](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1050561219
@chamikaramj -- how's this code look? This is an improvement, that doesn't break any existing tests -- so, I think is OK to merge.
Tables are not created properly [ with clustering ] for STORAGE_API_AT_LEAST_ONCE or STORAGE_WRITE_API -- so those will be new tickets that I add. Again, since this is an improvement over existing functionality ( and does not break things ), it seems OK to merge. I might revisit to get those working later, but for now the tests are for 'DEFAULT', and 'FILE_LOADS'.
I will write some follow up tickets for:
a) document this functionality. at least in https://beam.apache.org/documentation/io/built-in/google-bigquery/ , but perhaps anywhere else suggested
b) get clustering working with STORAGE_API_AT_LEAST_ONCE
c) get clustering working with STORAGE_WRITE_API
...? Anything else top of mind ( what else would we want to see, but needn't be captured by this current improvement?
Also, would be curious to learn the implementation differences between STORAGE_API_AT_LEAST_ONCE and STORAGE_WRITE_API ... I wonder if solved for one, if that fixes both.
I see some related issues that I'd like to see addressed ahead of implementing any 'fix' or additional integration tests just for clustering, when using the storage write api --> ex:
https://issues.apache.org/jira/browse/BEAM-13747
and
https://issues.apache.org/jira/browse/BEAM-13753 / https://issues.apache.org/jira/browse/BEAM-13990
If you agree is fine, do squash/merge, or give me the LGTM. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1035866839
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036911868
I think Beam Jenkins is down currently: https://issues.apache.org/jira/browse/INFRA-22878
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1035861809
Java Run PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036462186
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r804851343
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
+ }
+
+ @Override
+ public TableDestination getDestination(ValueInSingleWindow<TableRow> element) {
+ return new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName), null, null, CLUSTERING);
+ }
+
+ @Override
+ public TableDestination getTable(TableDestination destination) {
+ return destination;
+ }
+
+ @Override
+ public TableSchema getSchema(TableDestination destination) {
+ return SCHEMA;
+ }
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringNoPartitionTableFunction() throws Exception {
+ String tableName = "weather_stations_clustered_table_function_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(
+ (ValueInSingleWindow<TableRow> vsw) ->
+ new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName),
+ null,
+ null,
+ CLUSTERING))
+ .withClustering(CLUSTERING)
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(CLUSTERING, table.getClustering());
Review comment:
ya, that makes sense to have. Rowcount super straightforward. Hash is better, but ... I don't see a great way to easily generate to put into the test ( get it the first time ). I'll think more on the approach today. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r804352117
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
Easiest way to verify ( without actually thinking much :-) ) is going to be removing these lines and see if things still work. Once I re-confirm the below (that also writing the data), I'll then try removing these lines to verify whether works without.
Off handedly, I think it is needed ( I think we default to v2 coder, and need an override ... but ... is great if this is already handled and setting manually not required ). If this turns out to be necessary, then that seems like a new issue which I can at least file a ticket - and can consider addressing after this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039645247
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1041062191
> You can just try running the same pipeline with three write methods (no need to add additional tests). For testing streaming, for Dataflow, just add the "--streaming" flag. The pipeline will end since the input is bounded but it still runs using the streaming runner.
@chamikaramj -- this is hurting my head for some reason ... Where would you suggest the pipelines run to test these things? I've only been thinking in terms of IntegrationTests, is there another way you'd suggest to test this ( as it sounds like you think a one-off is sufficient, as a sanity check rather than an ongoing testing need ). Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1038385043
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039357587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r806265266
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
It looks like this is needed -- from the 'test' ( by removing setting this manually ) the IT failed, but it succeeds if these lines (103-106) setting coder manually are in there. https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/930/testReport/junit/org.apache.beam.sdk.io.gcp.bigquery/BigQueryClusteringIT/testE2EBigQueryClusteringNoPartitionDynamicDestinations/ ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048491404
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048406175
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048485506
[](https://github.com/brucearctor)Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048485457
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1050478775
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1049123851
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 90de8f9. Consider uploading reports for the commit 90de8f9 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...90de8f9](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036676433
@chamikaramj -- on methods: I had imagined that this working for ANY method/way would suffice for the exiting ticket ( since an improvement over what was there, and then writing up a blog and notes accordingly for what is known to work ).
Do we have ITs that spin up a pubsub topic ( or a Kafka Cluster and topics ), publish messages to it, runs the pipeline to consume, and then verifies that the sink [ ex: table ] winds up as expected ( perhaps then tears such things down ). I have this sort of automation in the past, for testing actual data pipelines, but that seems out of scope of this particular issue and is easily a different ticket/issue.
Put another way and to make concrete: how are we running ongoing tests for unbounded data? Me manually testing/checking is one thing, having actual tests that regularly run is another ( and really the only real way to verify that would be meaningful ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1037454103
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039357587
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039644966
Run Spotless PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r806267456
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
Sg. Resolving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039855392
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036487393
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036487393
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **decrease** coverage by `27.72%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 39c059f. Consider uploading reports for the commit 39c059f to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
===========================================
- Coverage 74.61% 46.88% -27.73%
===========================================
Files 653 203 -450
Lines 81802 20073 -61729
===========================================
- Hits 61038 9412 -51626
+ Misses 19783 9661 -10122
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...ks/go/pkg/beam/runners/dataflow/dataflowlib/job.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2pvYi5nbw==) | `14.76% <0.00%> (-1.31%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/typex/class.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3R5cGV4L2NsYXNzLmdv) | `82.72% <0.00%> (-0.61%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/exec/fn.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbi5nbw==) | `70.27% <0.00%> (-0.45%)` | :arrow_down: |
| [sdks/go/pkg/beam/runners/dataflow/dataflow.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93Lmdv) | `5.21% <0.00%> (-0.05%)` | :arrow_down: |
| [sdks/go/pkg/beam/runners/direct/direct.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9kaXJlY3QuZ28=) | `1.79% <0.00%> (-0.02%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/exec/hash.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9oYXNoLmdv) | `34.95% <0.00%> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/xlangx/expand.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuZC5nbw==) | `0.00% <0.00%> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/clone.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L2Nsb25lLmdv) | `22.44% <0.00%> (ø)` | |
| ... and [467 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...39c059f](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036464983
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1037322119
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r804340096
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
Do you need to set this coder manually ? I thought we choose the appropriate coder when clustering is enabled.
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
+ }
+
+ @Override
+ public TableDestination getDestination(ValueInSingleWindow<TableRow> element) {
+ return new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName), null, null, CLUSTERING);
+ }
+
+ @Override
+ public TableDestination getTable(TableDestination destination) {
+ return destination;
+ }
+
+ @Override
+ public TableSchema getSchema(TableDestination destination) {
+ return SCHEMA;
+ }
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringNoPartitionTableFunction() throws Exception {
+ String tableName = "weather_stations_clustered_table_function_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(
+ (ValueInSingleWindow<TableRow> vsw) ->
+ new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName),
+ null,
+ null,
+ CLUSTERING))
+ .withClustering(CLUSTERING)
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(CLUSTERING, table.getClustering());
Review comment:
Can we also verify the data written ?
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
+ }
+
+ @Override
+ public TableDestination getDestination(ValueInSingleWindow<TableRow> element) {
+ return new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName), null, null, CLUSTERING);
+ }
+
+ @Override
+ public TableDestination getTable(TableDestination destination) {
+ return destination;
+ }
+
+ @Override
+ public TableSchema getSchema(TableDestination destination) {
+ return SCHEMA;
+ }
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringNoPartitionTableFunction() throws Exception {
+ String tableName = "weather_stations_clustered_table_function_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(
+ (ValueInSingleWindow<TableRow> vsw) ->
+ new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName),
+ null,
+ null,
+ CLUSTERING))
+ .withClustering(CLUSTERING)
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(CLUSTERING, table.getClustering());
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringNoPartitionDynamicDestinations() throws Exception {
+ String tableName =
+ "weather_stations_clustered_dynamic_destinations_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(new ClusteredDestinations(tableName))
+ .withClustering(CLUSTERING)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(CLUSTERING, table.getClustering());
Review comment:
Ditto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor removed a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor removed a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1035867172
run java postcommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036939609
> I think Beam Jenkins is down currently: https://issues.apache.org/jira/browse/INFRA-22878
That seems to explain it, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040749946
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040748332
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1049245247
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 093f7d0. Consider uploading reports for the commit 093f7d0 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...093f7d0](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head e8b71c7. Consider uploading reports for the commit e8b71c7 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...e8b71c7](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048507756
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1050269777
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head 27ec3d7. Consider uploading reports for the commit 27ec3d7 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...27ec3d7](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1039295499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040718829
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1040729482
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r807505429
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
+ }
+
+ @Override
+ public TableDestination getDestination(ValueInSingleWindow<TableRow> element) {
+ return new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName), null, null, CLUSTERING);
+ }
+
+ @Override
+ public TableDestination getTable(TableDestination destination) {
+ return destination;
+ }
+
+ @Override
+ public TableSchema getSchema(TableDestination destination) {
+ return SCHEMA;
+ }
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringNoPartitionTableFunction() throws Exception {
+ String tableName = "weather_stations_clustered_table_function_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(
+ (ValueInSingleWindow<TableRow> vsw) ->
+ new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName),
+ null,
+ null,
+ CLUSTERING))
+ .withClustering(CLUSTERING)
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(CLUSTERING, table.getClustering());
Review comment:
I implimented # of lines and expected bytes -- This seemed sufficient and was easier to implement than using bigquerymatcher.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r806267456
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
Sg. Resolving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
# [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16578](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7fa5387) into [master](https://codecov.io/gh/apache/beam/commit/b45dc7e84a84ea33ceecc92b4c308805a31a68b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b45dc7e) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 7fa5387 differs from pull request most recent head f044c13. Consider uploading reports for the commit f044c13 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16578 +/- ##
==========================================
+ Coverage 74.61% 74.66% +0.04%
==========================================
Files 653 655 +2
Lines 81802 82347 +545
==========================================
+ Hits 61038 61482 +444
- Misses 19783 19865 +82
- Partials 981 1000 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `78.26% <0.00%> (-6.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `84.89% <0.00%> (-5.82%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | `74.69% <0.00%> (-4.79%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | `51.01% <0.00%> (-3.78%)` | :arrow_down: |
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | `90.52% <0.00%> (-3.16%)` | :arrow_down: |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `85.54% <0.00%> (-2.75%)` | :arrow_down: |
| [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `88.59% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.26% <0.00%> (-2.29%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.40% <0.00%> (-1.33%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/apache/beam/pull/16578/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b45dc7e...f044c13](https://codecov.io/gh/apache/beam/pull/16578?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1041029728
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1050277340
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1050352890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048406102
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048998697
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048290443
Thanks, Cham. Even if hard, doesn't mean not worth doing. **My philosophy is that if it is important to test once before merging then should be part of ongoing tests....**, so will at least try that.
Though, it doesn't seem to make sense to test the STORAGE_API_AT_LEAST_ONCE , since the test might fail ( if configured to expect exactly-once ), even though the API would be working as 'expected'/advertised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1048445536
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1049009289
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1055934390
@chamikaramj -- LGTY?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r804343590
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
+ }
+
+ @Override
+ public TableDestination getDestination(ValueInSingleWindow<TableRow> element) {
+ return new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName), null, null, CLUSTERING);
+ }
+
+ @Override
+ public TableDestination getTable(TableDestination destination) {
+ return destination;
+ }
+
+ @Override
+ public TableSchema getSchema(TableDestination destination) {
+ return SCHEMA;
+ }
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringNoPartitionTableFunction() throws Exception {
+ String tableName = "weather_stations_clustered_table_function_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(
+ (ValueInSingleWindow<TableRow> vsw) ->
+ new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName),
+ null,
+ null,
+ CLUSTERING))
+ .withClustering(CLUSTERING)
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(CLUSTERING, table.getClustering());
Review comment:
Sure ... I had set a default table expiration of 5 days, to not unnecessarily store IT data. Currently that is 5 days -- do we think we should have a longer retention period?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1034394114
Ack. Will check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on a change in pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on a change in pull request #16578:
URL: https://github.com/apache/beam/pull/16578#discussion_r806265266
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryClusteringIT.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryClusteringIT";
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigqueryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Override
+ public @Nullable Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
Review comment:
It looks like this is needed -- from the 'test' ( by removing setting this manually ) the IT failed, but it succeeds if it is in there. https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/930/testReport/junit/org.apache.beam.sdk.io.gcp.bigquery/BigQueryClusteringIT/testE2EBigQueryClusteringNoPartitionDynamicDestinations/ ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036681205
Run Spotless PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036461339
run java postcommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036982872
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1036951499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brucearctor commented on pull request #16578: [BEAM-10652] Allow Clustering without Partition in BigQuery
Posted by GitBox <gi...@apache.org>.
brucearctor commented on pull request #16578:
URL: https://github.com/apache/beam/pull/16578#issuecomment-1017982233
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org